Image processing device, image processing method, program, and display device

ABSTRACT

An image processing device includes circuitry configured to perform an effect process on at least one 3D model of a plurality of 3D models generated from a plurality of viewpoint images captured from a plurality of viewpoints at different times.

This application claims the benefit of Japanese Priority PatentApplication JP 2018-217179 filed on Nov. 20, 2018, the entire contentsof which are incorporated herein by reference.

TECHNICAL FELD

The present technology relates to an image processing device, an imageprocessing method, a program, and a display device, and moreparticularly to, for example, an image processing device, an imageprocessing method, a program, and a display device capable of providingan easily viewable image.

BACKGROUND ART

There has been proposed a method of generating a strobe image showing asubject (image) captured at a plurality of times (for example, seePatent Document 1). The strobe image shows the subject at a plurality oftimes, which makes it possible to easily grasp the movement andtrajectory of the subject.

CITATION LIST

Patent Literature

[PTL 91]

JP 2007-259477A

SUMMARY TECHNICAL PROBLEM

For example, in particular, in a case of Generating a strobe image for asubject that appears in a long-time frame (sequence), the strobe imagemay be difficult to see.

The present technology has been made in light of such a situation, andis intended to provide an easily viewable image, for example, an easilyviewable strobe image or the like.

SOLUTION TO PROBLEM

An image processing device or a program according to an aspect of thepresent technology is an image processing device that includes circuitryconfigured to perform an effect process on at least one 3D model of aplurality of 3D models generated from a plurality of viewpoint imagescaptured from a plurality of viewpoints; and generate a 2D image inwhich the plurality of 3D models having undergone the effect process isviewed from a predetermined viewpoint.

An image processing method according to an aspect of the presenttechnology includes performing an effect process on at least one 3Dmodel of a plurality, of 3D models generated from a plurality ofviewpoint images captured from a plurality of viewpoints and generatinga 2D image in which the plurality of 3D models having undergone theeffect process is viewed from a predetermined viewpoint.

According to the image processing device, the image processing method,and the program of an aspect of the present technology, the effectprocess is performed on at least one 3D model of the plurality of 3Dmodels generated from the plurality of viewpoint images captured fromthe plurality of viewpoints and generating a 2D image in which theplurality of 3D models having undergone the effect process is viewedfrom a predetermined viewpoint.

A display device according to an aspect of the present technologyincludes circuitry configured to receive a 2D image obtained byperforming an effect process on at least one 3D model of a plurality of3D models generated from a plurality of viewpoint images captured from aplurality of viewpoints and generating a 2D image in which the pluralityof 3D models having undergone the effect process is viewed from apredetermined viewpoint.

According to the display device of an aspect of the present technology,the 2D image is obtained by performing effect process on at least one ofa plurality of 3D models generated from a plurality of viewpoint imagescaptured from a plurality of viewpoints and generating a 2D image inwhich the plurality of 3D models having undergone the effect process isviewed from a predetermined viewpoint.

Note that the image processing device and the display device may beindependent devices or internal blocks constituting one device.

Furthermore, the program can be provided by being transmitted via atransmission medium, or by being recorded on a non-transitorycomputer-readable medium.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of asembodiment of as image processing system to which the present technologyis applied.

FIG. 2 is a flowchart of an example of a free viewpoint image displayprocess of displaying a free viewpoint image performed by the imageprocessing system.

FIG. 3 is a diagram illustrating an example of an unnatural freeviewpoint image.

FIG. 4 is a diagram illustrating an example of a natural free viewpointimage.

FIG. 5 is a diagram illustrating an example of a strobe image generatedby selecting a generation frame for each predetermined number of framesin a strobe section.

FIG. 6 is a view schematically illustrating an example of viewpointimages from a plurality of viewpoints obtained by an image capturingunit 11.

FIG. 7 is a diagram describing a first determination method for determining the motion of a subject by a strobe image generation unit 13.

FIG. 7 is a diagram describing a first determination method fordetermining the motion of a subject by a strobe image generation unit13.

FIG. 9 is a diagram describing a second determination method fordetermining the motion of a subject by the strobe image generation unit13.

FIG. 10 is a diagram describing a third determination method fordetermining the motion of a subject by the strobe image generation unit13.

FIG. 11 is a diagram describing an example of selection of a generationframe by the strobe image generation unit 13.

FIG. 12 is a diagram illustrating an example of a strobe image generatedby shifting 3D models of a subject shown in a plurality of generationframes from original positions.

FIG. 13 is a diagram describing 3D models in a strobe image to besubjected to an effect process by the effect processing unit 14.

FIG. 14 is a diagram describing specific examples of effect processes.

FIG. 15 is a diagram illustrating an example of effect process in effectmode 1.

FIG. 16 is a diagram illustrating an example of a strobe image showing3D models of two subjects.

FIG. 17 is a diagram illustrating another example of effect process ineffect mode 1.

FIG. 18 is a diagram illustrating an example of effect process in effectmode 2.

FIG. 19 is a diagram illustrating an example of effect process in effectmode 3.

FIG. 20 is a diagram illustrating an example of effect process in effectmode 4.

FIG. 21 is a diagram illustrating an example of effect process in effectmode 5.

FIG. 22 is a diagram illustrating an example of effect process in effectmode 6.

FIG. 23 is a diagram illustrating an example of effect process in effectmodes 7 to 9.

FIG. 24 is a diagram illustrating an example of effect process in effectmode 10.

FIG. 25 is a block diagram illustrating a configuration example of atransmission system to which the image processing system is applied.

FIG. 26 is a flowchart of an example of a process by a transmissiondevice 101.

FIG. 27 is a flowchart of an example of a process by a reception device102.

FIG. 28 is a block diagram illustrating another configuration example ofa transmission system to which the image processing system is applied.

FIG. 29 is a flowchart of a first example of a process by thetransmission device 101.

FIG. 30 is a flowchart of a first example of a process by the receptiondevice 102.

FIG. 31 is a flowchart of a second example of a process by thetransmission device 101.

FIG. 32 is a flowchart of a second example of a process by the receptiondevice 102.

FIG. 33 is a block diagram illustrating a configuration example ofanother embodiment of an image processing system to which the presenttechnology is applied.

FIG. 34 is a flowchart of an example of a free viewpoint image displayprocess of displaying a free viewpoint image.

FIG. 35 is a block diagram illustrating a configuration example of atransmission system to which the image processing system is applied.

FIG. 36 is a diagram describing an effect process of changing the sizeof a 3D model.

FIG. 37 is a block diagram illustrating a figuration example of anembodiment of a computer to which the present technology is applied.

DESCRIPTION OF EMBODIMENTS

<Image Processing System to which the Present Technology is Applied>

FIG. 1 is a block diagram illustrating a configuration example of anembodiment of an image processing system to which the present technologyis applied.

In the image processing system illustrated in FIG. 1, free viewpointdata is generated from a captured image in a manner capable ofgenerating a free viewpoint image in which the appearance of a subjectin a three-dimensional space from a virtual viewpoint is reproduced.Then, the free viewpoint image of the subject viewed from the virtualviewpoint is generated and displayed from the free viewpoint data.

The image processing system illustrated in FIG. 1 includes an imagecapturing unit 11, a free viewpoint data generation unit 12, a strobeimage generation unit 13, an effect processing unit 14, a free viewpointimage generation unit 15, and a display unit 16.

The image capturing unit 11 includes at least a plurality of cameras,and photographs a subject from a plurality of viewpoints. For example,the plurality of cameras constituting the image capturing unit 11 isdisposed so as to surround the subject, and each of the cameras capturesthe subject from a viewpoint at a position at which the camera isdisposed. The two-dimensional (2D) images captured from the positions ofthe cameras, in other words, the viewpoint images (moving images) fromthe plurality of viewpoints as 2D images captured from the plurality ofviewpoints are supplied frame by frame from the image capturing unit 11to the free viewpoint data generation unit 12.

Here, the image capturing unit 11 can be provided with a plurality ofdistance measurement devices in addition to a plurality of cameras. Thedistance measurement devices can be disposed at the same positions(viewpoints) as the cameras, or can be disposed at different positionsfrom the cameras. Each of the distance measuring devices measures thedistance to the subject from the position (view point) where thedistance measurement device is disposed, and generates a depth imagewhich is a 2D image having a depth as information regarding the distanceas a pixel value. The depth image is supplied from the image capturingunit 11 to the free viewpoint data generation unit 12.

Note that, when no distance measurement device is provided in the imagecapturing unit 11, the distance to the subject is measured according tothe principle of triangulation using viewpoint images from twoviewpoints among viewpoint images of a plurality of viewpoints, therebyto generate a depth image.

The free viewpoint data generation unit 12 generates free viewpoint dataof a 3D image frame by frame from the viewpoint images and the depthimages from the plurality of viewpoints from the image capturing unit11.

Here, the free viewpoint data is data of a 3D image from which a freeviewpoint image can be generated. As free viewpoint data, for example, aset of the viewpoint images and the depth images from the plurality ofviewpoints from the image capturing unit 11 can be adopted as it is.Furthermore, as free viewpoint data, for example, a 3D model (or 3D dataincluding a background image or the like) or a set of 2D images anddepth images from a plurality of viewpoints can be adopted.

In a case of adopting the set of the viewpoint images and the depthimages from the plurality of viewpoints from the image capturing unit 11as free viewpoint data, the free viewpoint data generation unit 12supplies the set of the viewpoint images and the depth images from theplurality of viewpoints from the image capturing unit 11 as freeviewpoint data to the strobe image generation unit 13.

In a case of adopting a 3D model as free viewpoint data, the freeviewpoint data generation unit 12 per modeling by Visual Hull or thelike using the viewpoint images from the plurality of viewpoints and thedepth images from the plurality of viewpoints from the image capturingunit 11. Then, the free viewpoint data generation unit 12 generates the3D model of the subject shown in the viewpoint images, and supplies the3D model (3D data including the 3D model) to the strobe image generationunit 13 as free viewpoint data. Note that, when the viewpoints of thedepth images from the image capturing unit 11 are different from theviewpoints of the viewpoint images from the image capturing unit 11, thefree viewpoint data generation unit 12 uses the depth images from theplurality of viewpoints from the image capturing unit 11 to generatedepth images from the viewpoints of the viewpoint images from the imagecapturing unit 11.

In a case of adopting a set of 2D images and depth images as freeviewpoint data from a plurality of viewpoints, the free viewpoint datageneration unit 12 generates a 3D model of the subject shown in theviewpoint images as described above, and generates a set of 2D imagesand depth images of the 3D model viewed from a plurality of viewpoints(the same viewpoints as those of the cameras constituting the imagecapturing unit 11 or different viewpoints). Then, the free viewpointdata generation unit 12 supplies the set of 2D images and depth imagesfrom the plurality of viewpoints generated from the 3D model as freeviewpoint data to the strobe image generation unit 13.

Hereinafter, for simplification of the description, a 3D model (3D dataincluding the 3D model) will be adopted as the free viewpoint dataunless otherwise specified.

Note that the amount of free viewpoint data can be reduced by adoptingnot 3D model but a set of 2D images and depth images from a plurality ofviewpoints generated from the 3D model as the free viewpoint data. WO2017/082076 presented by the present applicant describes a technique forgenerating and transmitting a set of 2D images and depth images from aplurality of viewpoints from a 3D model. In a case of generating a setof 2D images and depth images from a plurality of viewpoints from a 3Dmodel, the set of 211 images and depth images from the plurality ofviewpoints can be encoded by, for example, a coding method targeted for2D images such as multiview and depth video coding (MVCD), advancedvideo coding (AVC), or high efficiency video coding (HEVC), for example.

Here, the 3D model (the expression form thereof) can be roughly dividedinto a model called view independent (hereinafter also called VI model)and a model called view dependent (hereinafter also called VD model).

The VD model is a 3D model in which a 3D shape model as information of athree-dimensional shape and information of an image to be a texture areseparated. In the VD model, a 3D shape model is colored by mapping(texture mapping) an image to be a texture. According to the VD model,it is possible to express the degree of reflection on the surface of thesubject as a VD model different depending on the (virtual) viewpoint.

The VI model is a 3D model in which polygons and points as constituentelements of the 3D shape model have color information. Examples of a VImodel include a colored point cloud and a set of a 3D shape model and aDV map as color information of the 3D shape model, for example.According to the VI model, the colors possessed by polygons and pointscan be observed from any (virtual) viewpoint.

The strobe image generation unit 13 uses a 3D model as free viewpointdata of the 3D image from the free viewpoint data generation unit 12 togenerate free viewpoint data of the strobe image of the 3D image inwhich the 3D models of the same subject at a plurality of times (frames)are shown (arranged), and supplies the free viewpoint data to the effectprocessing unit 14.

Here, the strobe image is an image showing one or more same subjects(images) captured at a plurality of times. The strobe image that showsthe subject shown in the 2D image is also called 2D strobe image, andthe strobe image of the 3D image that shows the 3D model of the subjectis also called 3D strobe image. The strobe image generation unit 13generates a 3D strobe image. Here, the 3D image means an image spreadingthree-dimensionally, in other words, an image spreading in a depthdirection as well as in the horizontal and vertical directions.

Note that, in a case where free viewpoint data includes viewpoint imagesand depth images from a plurality of viewpoints, or 2D images and depthimages from a plurality of viewpoints, for generation of a strobe image,modeling is performed for each of a plurality of frames used for thegeneration to individually generate 3D models of the subject shown inthe plurality of frames. Then, the 3D models in the plurality of framesare combined with a background image (a three-dimensional space as thebackground image) to generate a strobe image. Otherwise, silhouetteimages of the subject shown in the plurality of frames are combined, andmodeling is performed using a composite silhouette image obtained by thecombination, thereby to generate a composite 3D model by combining 3Dmodels of the subject shown in the plurality of frames. Then, thecomposite 3D model is combined with the background image to generate astrobe image.

The effect processing unit 14 performs an effect process on the 3D modelseen in the strobe image in the free viewpoint data supplied from thestrobe image generation unit 13, and supplies the free viewpoint data ofthe strobe image having undergone the effect process to the freeviewpoint image generation unit 15.

Here, in the image processing system, a virtual viewpoint is setaccording to the user's operation, and is supplied to the effectprocessing unit 14, the free viewpoint image generation unit 15, andother necessary blocks. The effect processing unit 14 can perform aneffect process on 3D models at predetermined times among the 3D modelsat a plurality of times (frames) shown in the strobe image, for example,one or both 3D models in the past and future with reference to areference 3D model at a time when the latest virtual viewpoint is set.As the reference 3D model, instead of the 3D model of the time at whichthe latest virtual viewpoint is set, a 3D model specified by the usercan be adopted.

Note that the effect processing unit 14 can supply the free viewpointdata from the strobe image generation unit 121 to the free viewpointimage generation unit 15 without performing an effect process, forexample, according to the user's operation or the like. Furthermore, thestrobe image generation unit 13 can supply the free viewpoint data fromthe free viewpoint data generation unit 12 to the effect processing unit14 without generating a strobe image, for example, according to theuser's operation or the like. When the strobe image generation unit 13does not generate a strobe image, the effect processing unit 14 suppliesthe free viewpoint data from the strobe image generation unit 121 to thefree viewpoint image generation unit 15 without performing an effectprocess.

The free viewpoint image generation unit 15 uses the free viewpoint datafrom the effect processing unit 14, for example, to generate 2D images(data) in which a three-dimensional space captured by the imagecapturing unit 11 is viewed from virtual viewpoints such as 2D images inwhich a three-dimensional space shown in the strobe image in which the3D model has undergone the effect process is viewed from virtualviewpoints (here, including a set of a left-eye 2D image and a right-eye2D image) as free viewpoint images (data), and supplies the freeviewpoint images to the display unit 16.

The display unit 16 includes, for example, a 2D head-mounted display, a2D monitor, a 3D head-mounted display, a 3D monitor, and the like, anddisplays the free viewpoint image from the free viewpoint imagegeneration unit 15. A 3D head-mounted display or monitor is, forexample, a display device that presents stereoscopic vision bydisplaying a left-eye 2D image and a right-eye 2D image.

Note that the image processing system can be formed, for example, from aserver client system including a client, a cloud server, and the like.In this case, some or all of the free viewpoint data generation unit 12to the free viewpoint image generation unit 15 can be provided in thecloud server. The client can be provided with the remainder of the freeviewpoint data generation unit 12 to the free viewpoint image generationunit 15, and the display unit 16. The image capturing unit 11 can bedisposed at an arbitrary place, and the viewpoint image and the likeoutput by the image capturing unit 11 can be transmitted to the freeviewpoint data generation unit 12.

According to the image processing system configured as described above,for example, scenes of various sports such as soccer, rugby, baseball,wrestling, boxing, judo, golf, tennis, and gymnastics are captured asviewpoint images, and a strobe image showing a 3D model of a specificsubject such as a specific player can be generated. In this case, thestrobe image showing a 3D model of a specific player can be used forsports analysis such as analysis of the motion of that specific player.

FIG. 2 is a flowchart of an example of a free viewpoint image displayprocess of displaying a free viewpoint image performed by the imageprocessing system illustrated in FIG. 1.

In free viewpoint image display process, in step S11, the imagecapturing unit 11 photographs a subject from a plurality of viewpoints,and obtains viewpoint images and depth images from the plurality ofviewpoints on a frame-by-frame basis. The image capturing unit 11supplies the viewpoint images and depth images from the plurality ofviewpoints to the free viewpoint data generation unit 12, and theprocess proceeds from step S11 to step S12.

In step S12, the free viewpoint data generation unit 12 uses theviewpoint images and depth images from the plurality of viewpoints fromthe image capturing unit 11 to perform modeling of the subject shown inthe viewpoint images, thereby to generate a 3D model of the subject orthe like as free viewpoint data, for example, on a frame-by-frame basis.The free viewpoint data generation unit 12 supplies the 3D model of thesubject (and 3D data including the background image) as free viewpointdata to the strobe image generation unit 13, and the process proceeds tostep S13.

In step S13, the strobe image generation unit 13 determines the motionof the subject that is a 3D model as the free viewpoint data from thefree viewpoint data generation unit 12, and the process proceeds to stepS14.

In step S14, the strobe image generation unit 13 determines whether togenerate a strobe image.

Here, the determination as to whether to generate a strobe image in stepS14 is made, for example, according to the motion of the subjectdetermined in step S13. When the subject makes no motion, a strobe imageshowing 3D models of the subject at a plurality of times with no motionmay be difficult to view because the 3D models of the subject at aplurality of times are shown at substantially the same position.Therefore, in step S14, in a case where the subject makes no motion, itis determined that the strobe image is not to be generated, and in acase where the subject makes any motion, it is determined that thestrobe image is to be generated.

Note that the determination as to whether to generate a strobe image instep S14 can be made according to, for example, the user's operation.

When it is determined in step S14 that a strobe image is not to begenerated, the strobe image generation unit 13 and the effect processingunit 14 supply free viewpoint data to the free viewpoint imagegeneration unit 15 without performing processing. Then, the processproceeds from step S14 to step S19 skipping steps S15 to S18.

In this case, step S19, the free viewpoint image generation unit 15 usesthe free viewpoint data from the effect processing unit 14 to generate,as free viewpoint images, 2D images of the 3D models as free viewpointdata viewed from the virtual viewpoints. Then, the free viewpoint imagegeneration unit 15 supplies the free viewpoint images to the displayunit 16, and the process proceeds from step S19 to step S20.

In step S20, the display unit 16 displays the free viewpoint images fromthe free viewpoint image generation unit 15. In this case, the displayunit 16 displays images showing the 3D model of the subject viewed fromthe virtual viewpoint.

On the other hand, when it is determined in step S14 that a strobe imageis to be generated, the process proceeds to step S15.

In step S15, the strobe image generation unit 13 selects a frame(hereinafter also referred to as a generation frame) to be used forgeneration of a strobe image from among the frames of the 3D modelsupplied from the free viewpoint data generation unit 12, and theprocess proceeds to step S16.

Here, in the generation of a strobe image, in a frame sequence of aviewpoint image showing a subject to be a 3D model, the first frame(time) and the last frame of the subject that can be shown in the strobeimage are set in response to a user's operation and the like. Assumingthat the section from the first frame to the last frame of the subjectthat can be seen in the strobe image is a strobe section, if all framesof the strobe section are used as generation frames for generating astrobe image, the strobe image may become difficult to view because thesame number of 3D models of the same subject as the number of frames ofthe strobe section overlap.

Therefore, the strobe image generation unit 13 selects several frames asgeneration frames from the frames of the strobe section, and uses thegeneration frames (the 3D models of the subject shown in the generationframes) to generate the strobe image (free viewpoint data).

For example, the strobe image generation unit 13 can select, asgeneration frames, frames in which the degree of interference betweenthe 3D models is equal to or less than a threshold from the frames ofthe strobe section, for example. In other words, the strobe imagegeneration unit 13 calculates the degree of interference indicating thedegree of overlap between the 3D models of the subject in thethree-dimensional space in which the subject is shown in the frames ofthe strobe section. The degree of interference is calculated, forexample, as 100% in a case where 3D models in two arbitrary framescompletely overlap in the three-dimensional space, and as 0% in a casewhere the 3D models do not overlap at all. Then, the strobe imagegeneration unit 13 selects frames of which the degree of interference isequal to or less than a predetermined threshold, such as 0 to 10%, asgeneration frames. As described above, selecting frames of which thedegree of interference between the 3D models is equal to or less than athreshold as generation frames from the frames of the strobe section andgenerating a strobe image showing the 3D models in the generation framesmakes it possible to suppress the strobe image from becoming difficultto view with the 3D models overlapping.

Note that, in the selection of generation frames, for example, frames ofthe strobe section can be simply selected for each predetermined numberof frames.

In step S16, the strobe image generation unit 13 generates a strobeimage in which 3D models in a plurality of generation frames selectedfrom the frames of the strobe section are shown in a background image ofa three-dimensional space showing the subject of the 3D models. Then,the strobe image generation unit 13 supplies the strobe image to theeffect processing unit 14, and the process proceeds from step S16 tostep S17. Here, when only one subject is shown in a plurality ofgeneration frames, the strobe image generation unit 13 generates astrobe image showing the 3D model of the one subject. Furthermore, whena plurality of subjects is shown in a plurality of generation frames,the strobe image generation unit 13 can generate a strobe image showinga 3D model of each of the plurality of subjects. However, when aplurality of subjects is shown in a plurality of generation frames, the,strobe image generation unit 13 can generate a strobe image showing 3Dmodels of one or two or more subjects specified by the user, forexample, among the plurality of subjects shown in the plurality ofgeneration frames.

In step S17, the effect processing unit 14 determines whether to performan effect process on The 3D models of the strobe image from the strobeimage generation unit 13 (3D models shown in the strobe image). Thedetermination as to whether to generate a strobe image in step S17 canbe made, for example, according to the user's operation.

When it is determined in step S17 that an effect process is not to beperformed, the effect processing unit 14 supplies the strobe image fromthe strobe image generation unit 13 to the free viewpoint imagegeneration unit 15 without performing an effect process. Then, theprocess proceeds from step S17 to step S19 skipping step S18.

In his case, step S19, the free viewpoint image generation unit 15generates, as free viewpoint images, 2D images in which the strobe imagefrom the effect processing unit 14 is viewed from virtual viewpoints.Then, the free viewpoint image generation unit 15 supplies the freeviewpoint images to the display unit 16, and the process proceeds fromstep S19 to step S20.

In step S20, the display unit 16 displays the free viewpoint images fromthe free viewpoint image generation unit 15. In this case, the displayunit 16 displays 2D images showing 3D models of the subject in theplurality of generation frames viewed from virtual viewpoints (2D imagesin which the 3D strobe image is viewed from virtual viewpoints).

On the other hand, when it is determined in step S17 that an effectprocess is to be performed, the process proceeds to step S18.

In step S18, the effect processing unit 14 performs an effect processon, among the 3D models at a plurality of times (generation frames)shown in the strobe image from the strobe image generation unit 13, 3Dmodels in either or both of the past and future with reference to areference 3D model at a time when the latest virtual viewpoint is set.Then, the effect processing unit 14 supplies the strobe image havingundergone the effect process (showing the 3D models) to the freeviewpoint image generation unit 15, and the process proceeds from stepS18 to step S19.

In this case, in step S19, the free viewpoint image generation unit 15generates, as a free viewpoint image, 2D images in which the strobeimage after the effect process from the effect processing unit 14 isviewed from virtual viewpoints. Then, the free viewpoint imagegeneration unit 15 supplies the free viewpoint images to the displayunit 16, and the process proceeds from step S19 to step S20.

In step S20, the display unit 16 displays the free viewpoint images fromthe free viewpoint image generation unit 15. In this case, the displayunit 16 displays the 2D images in which the 3D models of the subject inthe plurality of generation frames viewed from the virtual viewpointsare shown and some of the 3D models have undergone the effect process(the 2D images in which the 3D strobe image having undergone the effectprocess is viewed from the virtual viewpoints).

As described above, performing an effect process on the 3D model makesit possible to provide an easily viewable image. In particular, forexample, performing an effect process on some or all of the 3D models ofthe same subject at a plurality of times shown in the strobe image makesit possible to provide an easily viewable strobe image. Note that, here,for the ease of understanding the description, a (3D) strobe image isgenerated, and then an effect process is performed on 3D models shown inthe strobe image. However, the generation of a strobe image and theexecution of the effect process on 3D models shown in the strobe imagecan be performed in parallel or in order changed as appropriate. Forexample, in the image processing system, after the effect process on 3Dmodels, a strobe image showing the 3D models having undergone the effectprocess can be generated.

<Generation of a Strobe Image>

FIG. 3 is a diagram illustrating an example of an unnatural freeviewpoint image.

FIG. 3 illustrates an example of a free viewpoint image that isgenerated from a (3D) strobe image generated using five frames asgeneration frames among the frames of a viewpoint image showing a ballas a subject rolling from the near to far sides.

In FIG. 3, the 3D models of the ball shown in the five generation framesare arranged (rendered) so as to give priority to the later 3D models.Therefore, the later (ball) 3D models are arranged to hide the earlier3D models in spite of being located on the near side. As a result, thefree viewpoint image illustrated in FIG. 3 is an unnatural image.

FIG. 4 is a diagram illustrating an example of a natural free viewpointimage.

FIG. 4 illustrates an example of a free viewpoint image that isgenerated from a (3D) strobe image generated using five frames asGeneration frames among the frames of a viewpoint image showing a ballas a subject rolling from the near to far sides.

In FIG. 4, the 3D models of the ball shown in the five generation framesare arranged to give priority to the 3D models on the near side.Therefore, the. 3D models on the near side are arranged to hide the 3Dmodels on the far side, in other words, the 3D models on the near sideare displayed on a priority basis. As a result, the free viewpoint imageis a natural image.

The strobe image generation unit 13 generates a strobe image usingdepths in which the 3D models on the near side are shown in the freeviewpoint image on a priority basis as described above.

FIG. 5 is a diagram illustrating an example of a strobe image generatedby selecting a generation frame for each predetermined number of framesin a strobe section.

FIG. 5 illustrates an example of a strobe image generated using eightframes as Generation frames among the frames of a viewpoint imageshowing a bail as a subject rolling from the near to far sides.

In the case of selecting a generation frame for each predeterminednumber of frames in the strobe section and generating a strobe imageusing the generation frames, when the moving speed of the subjectchanges, the distance between the 3D models of the subject shown in thestrobe image changes. For example, as illustrated in FIG. 5, when themoving speed of the subject decreases from a certain speed, the distancebetween the 3D models becomes narrow, and the degree of overlappingbetween the 3D models becomes large, which may make the strobe imagedifficult to view.

As described above with reference to FIG. 2, selecting frames of whichthe degree of interference between 3D models is equal to or less than athreshold from the frames of the strobe section makes the distancebetween the 3D models of the subject narrow with a change in the movingspeed of the subject, thereby suppressing the strobe image from becomingdifficult to view.

Note that, whether to select the frames of which the degree ofinterference between 3D models among the frames in the strobe section asgeneration frames or selecting a frame for each predetermined number offrames as a generation frame can be set according to the user'soperation, for example.

<Viewpoint Image>

FIG. 6 is a view schematically illustrating an example of viewpointimages from a plurality of viewpoints obtained by the image capturingunit 11.

Referring to FIG. 6, the image capturing unit 11 includes six camerasthat are arranged to surround a person as a subject. The cameras can bearranged around the subject or on the ceiling. The six camerassynchronously photograph the subject, and each of the cameras outputs a2D image obtained as a result of the photographing as a viewpoint imagefrom the position of the camera as viewpoint vp#i. The viewpoint vp#i isthe position of the i-th camera among the six cameras constituting theimage capturing unit 11.

FIG. 6 illustrates eight frames (times) of viewpoint images from sixviewpoints vp1 to vp6 output from the six cameras.

Besides the viewpoint images (frames) from the six viewpoints vp1 to vp6as described above, for example, the free viewpoint data generation unit12 generates 3D models of the subject shown in the viewpoint imagesusing depth images from the six viewpoints vp1 to vp6 and cameraparameters of the six cameras constituting the image capturing unit 11.[0086]

In other words, the free viewpoint data generation unit 12 obtains asilhouette image of the subject shown in the viewpoint image from theviewpoint vp#i, using a difference in foreground and background of theviewpoint image from the viewpoint vp#i, for example. Then, using thesilhouette image from the viewpoint vp#i, the viewpoint image and thedepth image from the viewpoint vp#i, and the camera parameters, the freeviewpoint data generation unit 12 performs modeling of the subject shownin the viewpoint image by Visual Hull or the like to generate a 3D modelof the subject.

Here, the camera parameters of the six cameras constituting the imagecapturing unit 11 include information such as the focal lengths of thecameras, the positional relationship between the cameras, the posturesof the cameras, and the distortions of the lenses included in thecameras.

Furthermore, the difference in foreground and background for obtainingthe silhouette image can be determined by taking the difference betweenthe background of the viewpoint image from the viewpoint vp#i and theviewpoint image from the viewpoint vp#i. The background of the viewpointimage from the viewpoint vp#i can be generated by photographing athree-dimensional space in the absence of the subject, or by using aplurality of frames of viewpoint images from the viewpoint vp#idifferent in the position of the subject.

<Determination on the Motion of a Subject>

FIGS. 7 and 8 are diagrams describing a first determination method fordetermining the motion of a subject by the strobe image generation unit13.

FIG. 7 illustrates a case where it is determined that the subject ismoving by the first determination method. FIG. 8 illustrates a casewhere it is determined that the subject is not moving by the firstdetermination method.

Referring to FIGS. 7 and 8, a skater as the subject is sliding in askating rink. When having an active sensor such as a time of fright(TOF) sensors or a light detection and ranging (LiDAR) sensor as adistance measuring device in addition to a plurality of cameras, theimage capturing unit 11 can determine the movement of the subjectaccording to a distance d#j to the subject measured by the activesensor. The distance d#j represents the distance to the subject measuredby the j-th active sensor among a plurality of active sensors.

Referring to FIGS. 7 and 8, four active sensors are provided around theskating rink. In the determination of the movement of the subject, thestrobe image generation unit 13 compares distances d1, d2, d3, and d4measured at time (frame) t by the four active sensors to distances d1,d2, d3 and d4 measured at a time different from time t, for example,time t′ after time t.

Then, as illustrated in FIG. 7, when one or more of the distances d1 tod4 differ (change) by a predetermined threshold or more between time tand time t′, the strobe image generation unit 13 determines that thereis movement of the subject.

On the other hand, as illustrated in FIG. 8, when none of the distancesd1 to d4 differs by a predetermined threshold or more between time t andtime t′, the strobe image generation unit 13 determines that there is nomovement of the subject.

Here, although the four active sensors are provided referring to FIGS. 7and 8, four or more active sensors may be provided, or one active sensormay be provided.

FIG. 9 is a diagram describing a second determination method fordetermining the motion of a subject by the strobe image generation unit13.

The second determination method is one of methods for determining themovement of the subject by the image capturing unit 11 without an activesensor.

In the second determination method, the strobe image generation unit 13reduces the number of frames in a predetermined section such as thestrobe section, for example, of a viewpoint image captured by any of thecameras constituting the image capturing unit 11 in such a manner as toleave several frames. Moreover, the strobe image generation unit 13 setsthe several frames left after the frame reduction as determinationframes for use in the determination of the movement of the subject, andgenerates silhouette images of the determination frames. Referring toFIG. 9, silhouette images are generated with five frames asdetermination frames.

The strobe image generation unit 13 detects an overlap between thesilhouette images of two arbitrary frames out of the plurality ofdetermination frames. Then, when there is no overlap between thesilhouette images in any one or more combinations of two determinationframes, for example, the strobe image generation unit 13 determines thatthere is movement of the subject.

On the other hand, when there is an overlap between the silhouetteimages in all the combinations of two determination frames, for example,the strobe image generation unit 13 determines that there is no movementof the subject.

FIG. 9A illustrates a case where there is no overlap between silhouetteimages in all combinations of any two frames among a plurality ofdetermination frames. FIG. 9B illustrates a case where there is anoverlap between the silhouette images in all combinations of any twoframes among the plurality of determination frames.

As illustrated in FIG. 9A, when there is no overlap between thesilhouette images, it is determined that there is movement of thesubject, and as illustrated in FIG. 9B, when there is an overlap betweenthe silhouette images, it is determined that there is no movement of thesubject.

FIG. 10 is a diagram describing a third determination method fordetermining the motion of a subject by the strobe image generation unit13.

The third determination method is another one of methods for determiningthe movement of the subject by the image capturing unit 11 without anactive sensor.

Assuming that the viewpoint images are captured through perspectiveprojection by (the cameras of) the image capturing unit 11, in theviewpoint images, the subject on the far side is shown in a small size,and the subject on the near side is shown in a large size. Therefore,for example, when the subject is moving from the near side to the farside, the silhouette images of the subject become smaller in size as thesubject moves to the far side, as illustrated in FIG. 10.

In the third determination method, the strobe image generation unit 13detects the sizes of silhouette images of a subject shown in a pluralityof determination frames. Then, for example, when the change in the sizeof the silhouette image is equal to or more than the threshold in anyone or more combinations of two determination frames, the strobe imagegeneration unit 13 determines that there is movement of the subject.

On the other hand, for example, when the change in the size of thesilhouette images is not equal to or more than the threshold value inall combinations of two determination frames, the strobe imagegeneration unit 13 determines that there is no movement of the subject.

Note that, according to the first to third determination methods, themovement of the subject can be determined before a 3D model of thesubject is generated. Otherwise, the determination on the movement ofthe subject can be made, for example, after generating a depth image orafter Generating a 3D model such as a point cloud or polygon, using thedepth image or position information regarding the position of the 3Dmodel a three-dimensional space.

Furthermore, when the subject is a person, feature points in theperson's face to be used for face detection are detected from theviewpoint image, and the movement of the subject can be determinedaccording to the positions of the feature points across frames in apredetermined section such as the strobe section.

Moreover, the motion vector of the subject is detected, and the movementof the subject can be determined according to the motion vector.

Furthermore, viewpoint images from some of viewpoints among viewpointimages from a plurality of viewpoints captured by the image capturingunit 11, for example, viewpoint images from four viewpoints captured byfour cameras among viewpoint images from 3D viewpoints captured by 3Dcameras are used to generate a simple 3D model of the subject shown inthe viewpoint images, and (the position of) the 3D model can be used todetermine the movement of the subject.

Moreover, in the image processing system, when a banding box surroundingone or more subjects shown in the viewpoint images is set, the movementof the subject can be determined according to the position of thebanding box across the frames of a predetermined section such as thestrobe section.

<Selection of Generation Frames>

FIG. 11 is a diagram describing an example of selection of a generationframe by the strobe image generation unit 3.

As described above with reference to FIG. 2, in order to suppress (thefree viewpoint images generated from) the strobe image from becomingdifficult to view due to an overlap between 3D models, the strobe imagegeneration unit 13 selects frames of which the degree of interferencebetween the 3D models is equal to or less than a threshold as generationframes from the frames of the strobe section, and can generate thestrobe image showing the 3D models of the generation frames.

FIG. 11A illustrates a strobe image of a skater sliding in a skatingrink with a threshold of 0%. Referring to FIG. 11A, since the thresholdis 0%, the 3D models of the subject at individual times (generationframes) do not overlap in the strobe image.

FIG. 11B illustrates a strobe image of a skater sliding in a skatingrink with a threshold of a small value that is greater than 0% (forexample, 10% or so). In FIG. 11B, since the threshold value is a valuegreater than 0%, in the strobe image, some of the 3D models of thesubject at individual times have somewhat overlap with adjacent 3Dmodels.

Here, the image processing system illustrated in FIG. 1 can be appliedto a transmission system that transmits free viewpoint data from thetransmission side to the reception side and generates and displays astrobe image on the reception side. In this case, it is possible totransmit an interference flag representing interference that is anoverlap between 3D models in a strobe image from the transmission sideto the reception side. Then, on the reception side, it is possible toGenerate a strobe image showing 3D models interfering (overlapping) or astrobe image showing 3D models not interfering according to theinterference flag. As the interference flag, for example, a 1-bit flagrepresenting the presence or absence of interference or a threshold ofthe degree of interference can be adopted.

Note that the image processing system can generate a strobe imageaccording to the user's operation even when (it is determined that)there is no movement of the subject. When there is no movement of thesubject, the degree of interference between the 3D models of the subjectshown in the frames of the strobe section may increase at any of theframes and may not fall below the threshold. Therefore, to generate astrobe image when there is no movement of the subject, it is possible toselect (a plurality of) generation frames for each predetermined numberof frames from the frames of the strobe section.

In this case, however, the strobe image in which the 3D models of thesubject shown in the plurality of generation frames are simply arrangedis difficult to see because the 3D models overlap largely.

Therefore, to generate a strobe image when there is no movement of thesubject, the strobe image generation unit 13 does not arrange the 3Dmodels of the subject (the subject at a plurality of times) at originalpositions (the positions of the subject in the three-dimensional space)but can arrange the 3D models shifted from the original positions suchthat the degrees of interference become equal to or less than thethreshold.

FIG. 12 is a diagram illustrating an example of a strobe image generatedby shifting 3D models of a subject shown in a plurality of generationframes from original positions.

FIG. 12 illustrates that a skater as the subject is spinning at thecenter of a skating rink, and the skater as the subject hardly changesin position.

In this case, the strobe image generation unit 13 can generate a strobeimage by shifting the 3D models of the subject (3D models at a pluralityof times) shown in the plurality of generation frames shifted from theoriginal positions such that the degrees of interference between the 3Dmodels become equal to or less than the threshold.

Referring to FIG. 12, a strobe image in which 3D models of the subjectshown in a plurality of generation frames are arranged in a circularshape in time order (of the generation frames), and a strobe image inwhich the 3D models are arranged in a linear shape are generated.

As described above, in the generation of a strobe image, shifting the 3Dmodels of the subject shown in the plurality of generation frames fromthe original positions such that the degrees of interference between the3D models becomes equal to or less than the threshold makes it possibleto suppress the strobe image from becoming difficult to view with the 3Dmodels overlapping largely.

<3D Models Targeted for Effect Process>

FIG. 13 is a diagram describing 3D models in a strobe image to besubjected to an effect process by the effect processing unit 14.

The effect processing unit 14 performs an effect. process on, among the3D models in a plurality of frames as a plurality of times selected fromthe frames of the strobe section, 3D models in either or both of thepast and future with reference to a reference 3D model at a time whenthe latest virtual viewpoint is set.

Target models as 3D models to be subjected to an effect process arespecified by an effect direction representing a time direction (pastdirection and future direction) with respect to the reference 3D modeland an effect distance representing the degree of separation from thereference 3D model.

As the effect direction, a past direction “past”, a future direction“future”, or both the past direction “past” and the future direction“future” can be set.

When the past direction “past” is set as the effect direction, an effectprocess is performed on the 3D models in the past direction from thereference 3D model. When the future direction “future” is set as theeffect direction, an effect process is performed on the 3D models in thefuture direction from the reference 3D model. When the past direction“past” and the future direction “future” are set as the effectdirection, an effect process is performed on the 3D models in the pastdirection and the 3D models in the future direction from the reference3D model.

The effect distance can be specified by the number of the 3D models“number”, the distance “distance”, or the time “time” from the reference3D model.

According to the number of models “number”, among 3D models shown in astrobe image, in other words, among 3D models shown in generation framesto be used to generate a strobe image, 3D models separated from thereference 3D model by the number of models “number” or more can bespecified as target models.

According to the distance “distance”, among 3D models shown in a strobeimage, 3D models separated from the reference 3D model by the distance“distance” or more can be specified as target models.

According to the time “time”, among 3D models shown in a strobe image,3D models separated from the reference 3D model by the time “time” ormore can be specified as target models.

The effect processing unit 14 performs an effect process on 3D modelsseparated from the reference 3D model in a strobe image by the number ofmodels “number”, the distance “distance”, or the time “time” or more, inthe past direction or the future direction or in the both past directionand future direction.

Hereinafter, for simplification of the description, it is assumed thatan effect process is performed on 3D models separated in the pastdirection from the reference 3D model, unless otherwise specified.

Here, when the strobe section is long and a large number of frames isselected as generation frames, a strobe image is generated using thelarge number of 3D models.

The strobe image generated using the large number of 3D models may bedifficult to view.

For example, in a strobe image generated using a large number of 3Dmodels, among 3D models of a predetermined subject shown in the strobeimage, 3D models preceding the reference 3D model by a specific time ormore may be a hindrance (for viewing) to following (future) 3D modelsand 3D models of other subjects.

Furthermore, in a strobe image generated using a number of 3D models, ina case where the subject moves along similar trajectories, for example,in a case where the subject performs a giant swing (backward swing orforward swing) on a horizontal bar, the temporally preceding (past) 3Dmodels and the temporally following 3D models have similar trajectories,which may make the time course difficult to understand.

Moreover, in a strobe image generated using a large number of 3D models,the data amount of the 3D models becomes large, and the amount ofprocessing required to display (free viewpoint images generated from)the strobe images becomes large.

The effect processing unit 14 performs an effect process on 3D modelsshown in a strobe image to make the strobe image easily viewable, andreduces the data amount of the strobe image and the amount of processingrequired to display the strobe image.

<Specific Examples of Effect Processes>

FIG. 14 is a diagram describing specific examples of effect processes.

Referring to FIG. 14, there are effect processes represented by effectmodes 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. In the effect modes 1 to 10,the effect direction and the effect distance described above withreference to FIG. 13 can be set.

Note that when the effect direction is not set, an effect process isperformed on the assumption that the past direction “past” is set as theeffect direction by default, for example.

The effect distance is specified by the number of the 3D models“number”, the distance “distance”, or the time “time” from the reference3D model as described above with reference to FIG. 13. For example, whenthe effect direction is set to the past direction “past” and the effectdistance is set to the number of models “number”=1, the effect processrepresented in the effect mode is performed on the target 3D modelsseparated from the reference 3D model by the number of models “number”=1or more in the past direction.

The effect mode 0 represents that no effect process is performed.

The effect mode 1 represents an effect process to transparentize 3Dmodels. In the effect process in the effect mode 1, the target modelscan be all transparentized at the same degree of transparency, or can begradually transparentized, that is, 3D models (target models) moreseparated in time or distance from the reference 3D model can betransparentized at higher degrees of transparency. How to transparentize3D models can be specified by, for example, a parameter defined inassociation with the effect mode 1. Note that, at the degree oftransparency of 100%, the target models are completely transparent. Inthis case, the result of the effect process in the effect mode 1 issubstantially similar to the effect mode 4 described later.

The effect mode 2 represents an effect process to cause 3D models todisappear gradually.

The effect mode 3 represents an effect process to reduce the number oftextures (the number of 2D images used as textures) of 3D models. in theeffect process in the effect mode 3, the numbers of textures of thetarget models can be all decreased to the same number, or can begradually decreased, that is, 3D models more separated in time ordistance from the reference 3D model can be more decreased in the numberof textures. How to decrease the number of textures of 3D models can bespecified by, for example, a parameter defined in association with theeffect mode 3.

The effect process in the effect mode 3 is to be performed on 3D modelsto be subjected to texture mapping, in other words, VD models, and isnot to be performed on VI models not to be subjected to texture mapping.

The effect mode 4 represents an effect process to erase 3D models.

The effect mode 5 represents an effect process to reduce at least one ofthe luminance and the saturation of 3D models. In the effect process inthe effect mode 5, the luminance and saturation of the target models canbe all reduced at the same ratio, or can be gradually reduced, that is,3D models more separated in time or distance from the reference 3D modelcan be reduced in the luminance and saturation at a higher ratio. How toreduce the luminance and saturation of 3D models and which of theluminance and saturation to be reduced can be specified by, for example,a parameter defined in association with the effect mode 5.

The effect mode 6 represents an effect process to limit the number of 3Dmodels shown in a strobe image. In the effect process in the effect mode6, the 3D models to be shown in the strobe image are limited to only the3D models other than the target models among the D models in thegeneration frames.

The effect mode 7 represents an effect process to turn 3D models intolow polygons, that is, an effect process to reduce the number of meshes(the number of polygons) of the 3D models. In the effect process in theeffect mode 7, the numbers of meshes of the target models can be alldecreased to the same number, or can be gradually decreased, that is, 3Dmodels more separated in time or distance from the reference 3D modelcan be more decreased in the number of meshes. How to decrease thenumber of meshes of 3D models can be specified by, for example, aparameter defined in association with the effect mode 7.

Note that the effect process in the effect mode 7 is performed on 3Dmodels formed from polygons, and is not performed on 3D models notformed from polygons, that is, wire frames, for example.

The effect modes 8 and 9 represent an effect process to change theexpression form of 3D models.

In other words, the effect mode 8 represents an effect process to change3D models formed from polygons into wire frames.

The effect mode 9 represents an effect process to change the expressionform of 3D models from view dependent to view independent, that is, aneffect process to change VD models to VI models (for example, pointclouds).

The effect mode 10 represents an effect process to erase 3D models whileleaving traces of the 3D models.

Although the effect direction and the effect distance can be set for theeffect modes 1 to 10, the default effect direction and effect distancecan be defined as needed.

For example, as the default effect direction in the effect modes 1 to10, the past direction “past” can be defined.

Furthermore, for example, as the default effect distance in the effectmode 1, the number of models “number”=1 can be defined.

In this case, when the effect direction and effect distance in theeffect mode 1 are not set, the effect process in the effect mode 1 isperformed on 3D models separated from the reference 3D model by one ormore models in the past direction, in other words, 3D models precedingthe 3D model next to the reference 3D model in the past direction.

Moreover, for example, as the default effect distance in the effect mode4, the distance “distance”=5 [m] can be defined.

In this case, when the effect direction and effect distance in theeffect mode 4 are not set, the effect process in the effect mode 4 isperformed on 3D models separated from the reference 3D model by 5 m ormore in the past direct on.

Furthermore, for example, as the default effect distance in the effectmode 5, time “time”=10 [sec] can be defined.

In this case, when the effect direction and effect distance in theeffect mode 5 are riot set, the effect process in the effect mode 5 isperformed on 3D models separated from the reference 3D model by 10seconds or more in the past direction.

Moreover, for example, as the default effect distance in the effect mode7, the number of models “number”=3 can be defined.

In this case, when the effect direction and effect distance in theeffect mode 7 are not set, the effect process in the effect mode 7 isperformed on 3D models separated from the reference 3D model by three ormore models in the past direction, in other words, 3D models precedingthe third 3D model from the reference 3D model in the past direction.

Note that a plurality of effect modes can be set for the effect processperformed by the effect processing unit 14. For example, when the effectmodes 1 and 3 are set for effect processes, effect processes areperformed to transparentize the 3D models and reduce the number oftextures.

Here, the image processing system illustrated in FIG. 1 can be appliedto a transmission system that transmits free viewpoint data from thetransmission side to the reception side and generates and displays astrobe image on the reception side. In this case, the effect mode andthe necessary effect direction and effect distance can be transmittedfrom the transmission side to the reception side as an effect flag(effect information) related to the effect process. Then, on thereception side, the effect process can be performed according to theeffect mode, the effect direction, and the effect distance as the effectflag.

FIG. 15 is a diagram illustrating an example of effect process in theeffect mode 1.

FIG. 15 illustrates an example of a strobe image showing 3D models of asubject moving in a straight line. Referring to FIG. 15, the strobeimage shows 3D models in five generation frames, and the 3D model of thelatest time (the 3D model in the latest generation frame) is a reference3D model. Then, the effect process in the effect mode 1 is performed onthe 3D models shown in the strobe image so that the 3D models precedingthe reference 3D model are gradually transparentized.

Therefore, according to the strobe image having undergone the effectprocess in the effect mode 1, the user can intuitively recognize thepassage of time. Moreover, the data amount of the strobe image can bereduced. Furthermore, it is possible to suppress the temporallypreceding (past) 3D models from becoming a hindrance (obstruction) tothe temporally following (future) 3D models, and provide an easilyviewable strobe image.

FIG. 16 is a diagram illustrating an example of a strobe image showing3D models of two subjects.

Referring to FIG. 16, 3D models of two subjects A and B at a pluralityof times (generation frames) are shown in a strobe image. Then, as inthe case of FIG. 15, the effect process in the effect mode 1 isperformed on the 3D models of the subject A so that the 3D modelspreceding the reference 3D model are gradually transparentized.

Referring to FIG. 16, the subject B moves so as to cross the trajectoryof the movement of the subject A. Therefore, the 3D models of thesubject B may overlap with the 3D models of the subject. A preceding thereference 3D model.

However, since the 3D models of the subject A preceding the reference 3Dmodel are gradually transparentized, even if the 3D models of thesubject B overlap with the 3D models of the subject A preceding thereference 3D model, it is possible to suppress the preceding 3D modelsfrom interfering with the 3D models of the subject B.

Note that, referring to FIG. 16, the effect process is performed on the3D models of the subject A but is not performed on the 3D models of thesubject B, but the effect process can be performed on both the 3D modelsof the subject A and the 3D models of the subject B.

FIG. 17 is a diagram illustrating another example of effect process inthe effect mode 1.

FIG. 17 illustrates an example of a strobe image showing 3D models of asubject moving in a circle. Referring to

FIG. 17, the effect process in the effect mode 1 is performed on the 3Dmodels shown in the strobe image so that the 3D models preceding thereference 3D model are gradually transparentized.

Here, in a case where the subject circles around the same trajectory,for example, in the strobe image showing the 3D models of the subject,the 3D models of the first round and the 3D models of the second roundmay be shown at the same time. When no effect process is performed, itis difficult to distinguish between the 3D models of the first round andthe 3D models of the second round in the strobe image, so that thestrobe image is difficult to view.

When the effect process in the effect mode 1 is performed, for example,if the 3D model at the latest time is the reference 3D model, the 3Dmodels shown in the strobe image becomes more transparent withdecreasing proximity to the reference 3D model. Therefore, it ispossible to provide an easily viewable strobe image that allows the userto easily distinguish between the 3D model of the first round and the 3Dmodel of the second round.

Furthermore, according to the effect process in the effect mode 1, in astrobe image showing 3D models of a subject circling along the sametrajectory, (the 3D models of) the subject is expressed such that the 3Dmodels of the latest round enter (jack in) the 3D models of the previousround. Therefore, it is possible to provide a free view video (FVV) thatallows the user to easily compare the subject of the latest round andthe subject of the previous round.

Note that, in a case of a subject circling the same trajectory, forexample, an athlete as the subject runs around a track, or a car or amotorbike as the subject races on a circuit, a gymnast as the subjectswings on a horizontal bar.

Here, the virtual viewpoint can be set at an arbitrary position. Thevirtual viewpoint can be set at a position that looks down on thesubject or at the position of the subject. When the virtual viewpoint isset to a position that looks down on the subject, the user can view (the3D models of) the subject from the standpoint of a third party. When thevirtual viewpoint is set to the position of the subject (first-personviewpoint), the user can view ((the 3D models of) the subject in thepast that has been transparentized by the effect process from theviewpoint of the subject as if he or she follows the subject.

FIG. 18 is a diagram illustrating an example of effect process in theeffect mode 2.

FIG. 18 illustrates an example of a strobe image showing 3D models of asubject moving in a straight line. Referring to FIG. 18, the strobeimage shows 3D models in five generation frames, and the 3D model of thelatest time is a reference 3D model. Then, the effect process in theeffect mode 2 is performed on the 3D models shown in the strobe image sothat the 3D models preceding the reference 3D model gradually (andgently) disappear.

Therefore, according to the strobe image having undergone the effectprocess in the effect mode 2, the user can intuitively recognize thepassage of time. Moreover, the data amount of the strobe image can bereduced. Furthermore, it is possible to suppress the temporallypreceding 3D models from becoming a hindrance to the temporallyfollowing 3D models, and provide an easily viewable strobe image.

Note that, when the 3D models are formed from polygons, the effectprocess in the effect mode 2 in which 3D models gradually disappear canbe performed, for example, by decreasing more greatly the number ofpolygons in the 3D models more separated from the reference 3D model.Furthermore, the effect process in the effect mode 2 in which 3D modelsgradually disappear can be performed, for example, by decreasing moregreatly the number of polygons in the 3D models more separated from thereference 3D model and, among the 3D models having been decreased in thenumber of polygons at a predetermined ratio from the original number ofpolygons, decreasing more greatly the 3D models more separated from thereference 3D model in the number of points (the number of points inpoint cloud). Moreover, the effect process in the effect mode 2 n which3D models gradually disappear can be performed, for example, by changingthe 3D models to point clouds and decreasing more greatly the number ofpoints in the 3D models more separated from the reference 3D model.

When the 3D models are formed from point clouds, the effect process inthe effect mode 2 in which 3D models gradually disappear can beperformed by decreasing more greatly the number of points in the 3Dmodels more separated from the reference 3D model.

According to the effect process in the effect mode 2 as described above,each of the 3D models separated from the reference 3D model isrepresented as a collection of sparse particles like a fog.

FIG. 19 is a diagram illustrating an example of effect process in theeffect mode 3.

FIG. 19 illustrates an example of a strobe image showing 3D models of asubject moving in a straight line. Referring to FIG. 19, the strobeimage shows 3D models in five generation frames, and the 3D model of thelatest time is a reference 3D model. Then, the effect process in theeffect mode 3 is performed on the 3D models shown in the strobe image sothat the 3D models preceding the reference 3D model are graduallydecreased in the number of textures.

Therefore, according to the strobe image having undergone the effectprocess in the effect mode 3, the texture quality of the 3D models isdegraded, but the data amount of the strobe image and the amount ofprocessing required to display the strobe image can be reduced.

Note that, when 3D models more separated from the reference 3D model aremore decreased in the number of textures in the effect process in theeffect mode 3, the rate of decreasing the number of textures can take avalue obtained by dividing 100% by the number of 3D models shown in thestrobe image, for example. In this case, for a strobe image showing 3Dmodels in five generation frames, the rate of decreasing the number oftextures is 20%=100/5. Therefore, in a case where the reference 3D modelshown in the strobe image is subjected to texture mapping with viewpointimages from 10 viewpoints, for example, the first 3D model in the pastdirection of the reference 3D model is subjected to texture mapping withviewpoint images from eight viewpoints decreased by 20% from the 10viewpoints of the texture mapping of the reference 3D model. The second3D model in the past direction of the reference 3D model is subjected totexture mapping with viewpoint images from six viewpoints reduced by 20%from the eight viewpoints of the texture mapping of the first 3D modelin the past direction. Hereinafter, similarly, the third 3D model in thepast direction of the reference 3D model is subjected to texture mappingwith viewpoint images from four viewpoints reduced by 20% from the sixviewpoints of the texture mapping of the second 3D model in the pastdirection, and the fourth 3D model in the past direction of thereference 3D model is subjected to texture mapping with the viewpointimages from two viewpoints reduced by 20% from the four viewpoints ofthe texture mapping of the third 3D model in the past direction.

FIG. 20 is a diagram illustrating an example of effect process in theeffect mode 4.

FIG. 20 illustrates an example of a strobe image showing 3D models of asubject moving in a straight line. Referring to FIG. 20, the strobeimage shows 3D models in five Generation frames, and the 3D model of thelatest time is a reference 3D model. Then, the effect process in theeffect mode 4 is performed on the 3D models shown in the strobe image sothat, among the 3D models preceding the reference 3D model, the fourth3D model from the reference 3D model as a target model, for example, isdisplayed once and then erased after a lapse of a certain time.

Therefore, according to the strobe image having undergone the effectprocess in the effect mode 4, the data amount of the strobe image andthe amount of processing required to display the strobe image can bereduced.

Note that, in the effect process in the effect mode 4, for example, thecertain time from the display to erasing of the target model can bespecified by a parameter defined in association with the effect mode 4.

Furthermore, in the effect process in the effect mode 4, instead oferasing the target model after a lapse of a certain time, the targetmodel can be erased when the number of the 3D models shown in the strobeimage has reached a certain number. The certain number of the 3D modelswith which to erase the target model can be specified by a parameterassociated with the effect mode 4, as in the case of the certain timedescribed above, for example. FIG. 21 is a diagram illustrating anexample of effect process in the effect mode 5.

FIG. 21 illustrates an example of a strobe image showing 3D models of asubject moving in a straight line. Referring to FIG. 21, the strobeimage shows 3D models in fire generation frames, and the 3D model of thelatest time is a reference 3D model. Then, the effect process in theeffect mode 5 is performed on the 3D models shown in the strobe image sothat the 3D models preceding the reference 3D model are graduallydecreased in luminance and saturation.

Therefore, according to the strobe image having undergone the effectprocess in the effect mode 5, the data amount of the strobe image andthe amount of processing required to display the strobe image can bereduced.

Note that, in a case of decreasing more greatly the luminance andsaturation of the 3D models more separated from the reference 3D modelin the effect process in the effect mode 5, the rate of reducing theluminance and saturation can be the same as the rate in the case in theeffect mode 3, for example. In other words, the rate of reducing theluminance and saturation can be a value obtained by dividing 100% by thenumber of 3D models shown in the strobe image. In this case, the rate ofdecreasing the luminance and saturation in a strobe image showing 3Dmodels in five generation frames is 20%=100/5. The 3D models precedingthe reference 3D model are decreased in luminance and saturation by 20%with distance by one each model from the reference 3D model.

Furthermore, instead of decreasing more greatly the luminance andsaturation of the 3D models with distance by one each model from thereference 3D model, the luminance and saturation of the 3D models can bedecreased with distance for a certain time from the reference 3D model.

FIG. 22 is a diagram illustrating an example of effect process in theeffect mode 6.

FIG. 22 illustrates an example of a strobe image showing 3D models of asubject moving in a straight line. Referring to FIG. 22, the strobeimage shows 3D models in five generation frames, and the 3D model justbefore the 3D model of the latest time is a reference 3D model. Then,the effect process in the effect mode 6 is performed on the 3D modelsshown in the strobe image, so that the 3D models shown in the strobeimage are limited to three models including the reference 3D model, inother words, the reference 3D model and the two 3D models adjacent toeach other in the past direction of the reference 3D model.

Therefore, according to the strobe image having undergone the effectprocess the effect mode 6, the data amount of the strobe image can bereduced. Furthermore, for example, it is possible to suppress thetemporally preceding 3D models from becoming a hindrance to thetemporally following 3D models, and provide an easily viewable strobeimage.

FIG. 23 is a diagram illustrating an example of effect process in theeffect modes 7 to 9.

FIG. 23 illustrates an example of a strobe image showing 3D models of asubject moving in a straight line. Referring to FIG. 23, the strobeimage shows 3D models in five generation frames, and the 3D model of thelatest time is a reference 3D model. Then, the effect process in any onein the effect modes 7 to 9 is performed on the 3D models shown in thestrobe image.

In other words, FIG. 23A illustrates a state in which the effect processin the effect mode 7 is performed on the 3D models shown in the strobeimage, so that one of the 3D models formed from polygons is decreased inthe number of meshes (the number of polygons).

FIG. 23B illustrates a state in which the effect process in the effectmode 8 is performed on the 3D models shown in the strobe image, so thatone of the 3D models formed from polygons is changed into a wire frame.

Note that, in the effect process in the effect mode 9, a 3D model as aVD model shown in a strobe image is changed to a VI model such as apoint cloud, but the illustration thereof is omitted.

According to the strobe image having undergone the effect process in theeffect modes 7 to 9, the data amount of the strobe image can be reduced.Furthermore, for example, it is possible to suppress the temporallypreceding 3D models from becoming a hindrance to the temporallyfollowing 3D models, and provide an easily viewable strobe image.

Note that, when it is required to further reduce the amount of data, forexample, in the effect process in the effect mode 9, after the 3D modelsare changed to point clouds, the number of points in the point cloudscan be decreased and the information on the colors of the points can bedecreased, although the quality of the shapes decreases. The sameapplies to VI models other than point clouds.

FIG. 24 is a diagram illustrating an example of effect process in theeffect mode 10.

FIG. 24 illustrates an example of a strobe image showing 3D models of asubject moving in a straight line. Referring to FIG. 24, the strobeimage shows 3D models in five generation frames, and the 3D model of thelatest time is a reference 3D model. Then, the effect process n theeffect mode 10 is performed on the 3D models shown in the strobe imageso that, among the 3D models preceding the reference 3D models, thethird and fourth 3D models from the reference 3D model are erased withshadows of these 3D models left as traces.

Therefore, according to the strobe image having undergone the effectprocess in the effect mode 10, the user can intuitively recognize thepassage of time. Moreover, the data amount of the strobe image can bereduced. Furthermore, it is possible to suppress the temporallypreceding (past.) 3D models from becoming a hindrance (obstruction) tothe temporally following (future) 3D models, and provide an easilyviewable strobe image.

Note that, as the trace of a 3D model, instead of the shadow, forexample, a point representing a position where the 3D model was present,a line representing a trajectory of movement of the 3D model, or thelike can be adopted.

Furthermore, in the effect process in the effect mode 10, the trace of a3D model can be left by storing the position of the 3D model in thethree-dimensional space and drawing a shadow or the like according tothe position.

<Transmission System>

FIG. 25 is a block diagram illustrating a configuration example of atransmission system to which the image processing system illustrated inFIG. 1 is applied.

Referring to FIG. 25, the transmission system includes a transmissiondevice 101 and a reception device 102. In the transmission system, freeviewpoint data is transmitted from the transmission device 101 to thereception device 102. Then, in the reception device 102, a strobe imageis generated using the free viewpoint data from the transmission device101 and is displayed.

The transmission device 101 includes an image capturing unit 111, a freeviewpoint data generation unit 112, an image processing unit 113, anencoding unit 114, and a transmission unit 115.

The image capturing unit 111 is configured in the same manner as theimage capturing unit 11 illustrated in FIG. 1, photographs a subjectfrom a plurality of viewpoints, and supplies viewpoint images from theplurality of viewpoints to the free viewpoint data Generation unit

As with the free viewpoint data generation unit 12 illustrated in FIG.1, the free viewpoint data generation unit 112 generates free viewpointdata using the viewpoint images from the plurality of viewpoints fromthe image capturing unit 111, for example, and supplies the freeviewpoint data to the image processing unit 113.

The image processing unit 113 includes a strobe image generation unit121 and an effect processing unit 122. The image processing unit 113uses the free viewpoint data from the free viewpoint data generationunit 112 to generate as necessary a strobe image (3D strobe image)showing 3D models of a subject at a plurality of times (frames) in theviewpoint images captured by the image capturing unit 111, and performsan effect process on the strobe image.

As with the strobe image generation unit 13 illustrated in FIG. 1, thestrobe image generation unit 121 uses the free viewpoint data from thefree viewpoint data generation unit 12 to generate a strobe imageshowing 3D models at a plurality of times, and supplies the freeviewpoint data of the strobe image to the effect processing unit 122.

As with the effect processing unit 14 illustrated in FIG. 1, the effectprocessing unit 122 performs an effect process on the 3D model seen inthe strobe image in the free viewpoint data supplied from the strobeimage generation unit 121, and supplies the free viewpoint data of thestrobe image having undergone the effect process to the encoding unit114.

Note that the image processing unit 113 can supply the free viewpointdata from the strobe image generation unit 121 to the encoding unit 114without the effect process by the effect processing unit 122 accordingto the user's operation or the like, for example. Moreover, the imageprocessing unit 113 can supply the free viewpoint data from the freeviewpoint data generation unit 112 to the encoding unit 114 without thegeneration of a strobe image at the strobe image generation unit 121 andthe effect process by the effect processing unit 122 according to theuser's operation or the like, for example. Furthermore, the imageprocessing unit 113 can performs generation of a strobe image and aneffect process in parallel, that is, can generate a (3D) strobe imageafter effect process in which an effect process has been performed on 3Dmodels from free viewpoint data.

The encoding unit 114 encodes the free viewpoint data supplied from (theeffect processing unit 122 of) the image processing unit 113 accordingto a predetermined encoding method, and supplies coded data obtained bythe encoding to the transmission unit 115.

The transmission unit 115 transmits the coded data from the encodingunit 114 by wired communication or wireless communication.

The reception device 102 includes a reception unit 131, a decoding unit132, a free viewpoint image generation unit 133, and a display unit 134.

The reception unit 131 receives the coded data transmitted from (thetransmission unit 115 of) the transmission device 101, and supplies thecoded data to the decoding unit 132.

The decoding unit 132 decodes the coded data from the reception unit 131into free viewpoint data according to the encoding method of theencoding unit 114, and supplies the free viewpoint data to the freeviewpoint image generation unit 133.

As with the free viewpoint image generation unit 15 illustrated in FIG.1, the free viewpoint image generation unit 133 uses the free viewpointdata from the decoding unit 132 to generate 2D images in which athree-dimensional space captured by the image capturing unit 111 isviewed from virtual viewpoints such as 2D images in which athree-dimensional space shown in a strobe image in which 3D models haveundergone an effect process is viewed from virtual viewpoints setaccording to the user's operation on the reception device 102, forexample, and supplies the 2D images as free viewpoint images to thedisplay unit 134.

As with the display unit 16 illustrated in FIG. 1, the display unit 134includes, for example, a 2D head-mounted display, a 2D monitor, a 3Dhead-mounted display, a 3D monitor, and the like, and displays the freeviewpoint image from the free viewpoint image generation unit 133.

FIG. 26 is a flowchart of an example of a process by the transmissiondevice 101 illustrated in FIG. 25.

In step S111, the image capturing unit 111 of the transmission device101 photographs a subject from a plurality of viewpoints, suppliesviewpoint images from the plurality of viewpoints obtained by thephotographing to the free viewpoint data generation unit 112, and theprocess proceeds to S112.

In step S112, the free viewpoint data generation unit 112 generates freeviewpoint data using the viewpoint images from the plurality ofviewpoints from the image capturing unit 111, for example.

For example, the free viewpoint. data generation unit 112 uses theviewpoint images from the plurality of viewpoints from the imagecapturing unit 111 to generate 3D models of the subject shown in theviewpoint images by Visual Hull or the like, thereby making it possibleto generate the 3D models (3D data including the 3D models and Thebackground image) as free viewpoint image data. The 3D model may be, forexample, a VD model haying a 3D shape model and viewpoint images of aplurality of viewpoints to be a texture, and for example, a coloredpoint cloud, or a VI model such as a set of a 3D shape model with a UVmap as information of a color of the 3D shape model.

Furthermore, for example, the free viewpoint data generation unit 112converts the 3D models of the subject shown in the viewpoint images,which is generated using the viewpoint images from the plurality ofviewpoints, into 2D images and depth images viewed from a plurality ofviewpoints (that may be the same as the viewpoints of the viewpointimages or may be different from the same), thereby making it possible togenerate the 2D images and the depth images from the plurality ofviewpoints as free viewpoint data. Note that the process of convertingthe 3D models into the 2D images and depth images from the plurality ofviewpoints can be performed by the free viewpoint data generation unit112 or by the encoding unit 114 before encoding the free viewpoint data,for example.

The free viewpoint data generation unit 112 supplies the free viewpoint.data to the strobe image generation unit 121 in the image processingunit 113, and the process proceeds from step S112 to step S113.

In step S113, the strobe image generation unit 121 determines whether togenerate a strobe image.

Here, the determination as to whether to generate a strobe image in stepS113 can be made, for example, according to the motion of the subject,as in step S14 illustrated in FIG. 2. Furthermore, the determination asto whether to generate a strobe image in step S113 can be made accordingto, for example, the user's operation on the transmission device 101.

When it is determined in step S113 that a strobe image is not to begenerated, the strobe image generation unit 121 and the effectprocessing unit 122 supply free viewpoint data to the encoding unit 114without performing processing. Then, the process proceeds from step S113to step S118 skipping steps S114 to S117.

Furthermore, when it is determined in step S113 that a strobe image isto be generated, the process proceeds to step S114.

In step S114, the strobe image generation unit 121 generates a strobeimage using the free viewpoint data from the free viewpoint datageneration unit 112.

In other words, the strobe image generation unit 121 selects a pluralityof generation frames from the frames of the free viewpoint data from thefree viewpoint data generation unit 112. Moreover, the strobe imagegeneration unit 121 generates a strobe image showing 3D models (3Dmodels at a plurality of times) of the subject shown in the plurality ofgeneration frames.

The strobe image generation unit 121 supplies free viewpoint data of thestrobe image to the effect processing unit 122, and the process proceedsfrom step S114 to step S115.

In step S115, the effect processing unit 122 determines whether toperform an effect process on the 3D models of the strobe image of whichthe free viewpoint data is supplied from the strobe image generationunit 121. The determination as to whether to generate a strobe image instep S115 can be made according to the user's operation on thetransmission device 101 and the like, for example.

When it is determined in step S115 that an effect process is not to beperformed, the effect processing unit 122 supplies the free viewpointdata of the strobe image from the strobe image generation unit 121 tothe encoding unit 114 without performing an effect process. Then, theprocess proceeds from step S115 to step S118 skipping steps S116 andS117.

Furthermore, when it is determined in step S115 that an effect processis to be performed, the process proceeds to step S116.

In step S116, the effect processing unit 122 sets, for example, theeffect flag (FIG. 14), that is, the effect mode, the effect direction,and the effect distance in accordance with the user's operation on thetransmission device 101 or the like, and the process proceeds to S117.

In step S117, according to the effect flag, the effect processing unit122 performs an effect process on the 3D models at a plurality of times(generation frames) shown in the strobe image of which the freeviewpoint data is supplied from the strobe image generation unit 121.

For example, in a case where the effect mode is set to effect mode 1,the effect direction is set to past, and the effect distance is set totime=10, the effect processing unit 122 performs an effect process totransparentize the 3D models separated by 10 seconds or more in the pastdirection from the reference 3D model in the strobe image.

Then, the effect processing unit 122 supplies the free viewpoint data ofthe strobe image having undergone the effect process to the encodingunit 114, and the process proceeds from step S117 to step S118.

In step S118, the encoding unit 114 encodes the free viewpoint datasupplied from the effect processing unit 122 and supplies coded dataobtained by the encoding to the transmission unit 115, and the processproceeds to step S119.

In step S119, the transmission unit 115 transmits the coded data fromthe encoding unit 114, and the process is terminated.

As described above, in the transmission device 101, an effect process isperformed in accordance with the effect flag set in response to theuser's operation on the transmission device 101 or the like. Therefore,the user of the transmission device 101 can apply a desired effect tothe 3D models of a subject shown in the strobe image by setting theeffect flag.

FIG. 27 is a flowchart of an example of a process by the receptiondevice 102 illustrated in FIG. 25.

In step S131, the reception unit 131 of The reception device 102receives the coded data transmitted from the transmission unit 115 ofthe transmission device 101 and supplies the coded data to the decodingunit 132, and the process proceeds to step S132.

In step S132, the decoding unit 132 decodes the coded data from thereception unit 131 and supplies free viewpoint data obtained by thedecoding to the free viewpoint image generation unit 133, and theprocess proceeds to step S133. Here, when the free viewpoint dataobtained by the decoding is 2D images and depth images from a pluralityof viewpoints, the decoding unit 132 can convert the 2D images and depthimages from the plurality of viewpoints into 3D models, and supplies the3D models (3D data including the 3D models) as free viewpoint data tothe free viewpoint image generation unit 133.

In step S133, the free viewpoint image generation unit 133 uses the freeviewpoint data from the decoding unit 132 to generate, as free viewpointimages, 2D images in which a three-dimensional space captured by theimage capturing unit 111 of the transmission device 101 as viewed from avirtual viewpoint set by the user's operation on the reception device102 or the like. In other words, for example, the free viewpoint imagegeneration unit 133 generates 2D images in which a three-dimensionalspace shown in a strobe image obtained by performing an effect processon 3D models is viewed from a virtual viewpoint, as free viewpointimages. Then, the free viewpoint image generation unit 133 supplies thefree viewpoint image (data) to the display unit 134, and the processproceeds from step S133 to step S134.

In step S134, the display unit 134 displays the free viewpoint imagesfrom the free viewpoint image generation unit 133, and the process isterminated.

As described above, in the reception device 102, the free viewpointimages generated using free viewpoint data obtained by decoding thecoded data from the transmission device 101 are displayed. Therefore,when a strobe image is generated and an effect process performed on thesubject shown in the strobe image in the transmission device 101, theuser of the reception device 102 can view the strobe image in which theeffect is applied to the 3D models (the free viewpoint images generatedfrom the strobe image).

FIG. 28 is a block diagram illustrating another configuration example ofa transmission system to which the image processing system illustratedin FIG. 1 is applied.

Note that the components corresponding to those illustrated in FIG. 25are denoted with the same reference numerals, and the descriptionthereof will be appropriately omitted below.

Referring to FIG. 28, the transmission system includes a transmissiondevice 101 and a reception device 102. The transmission device 101includes an image capturing unit 111, a free viewpoint data generationunit 112, an encoding unit 114, and a transmission unit 115. Thereception device 102 includes a reception unit 131, a decoding unit 132,a free viewpoint image generation unit 133, a display unit 134, and animage processing unit 141.

Therefore, the transmission system illustrated in FIG. 28 is in commonwith the transmission system illustrated in FIG. 25, in that thetransmission system includes the transmission device 101 and thereception device 102, the transmission device 101 includes the imagecapturing unit 111, the free viewpoint data generation unit 112, theencoding unit 114, and the transmission unit 115, and the receptiondevice 102 includes the reception unit 131, the decoding unit 132, thefree viewpoint image generation unit 133, and the display unit 134.

However, the transmission system illustrated in FIG. 28 is differentfrom the transmission system illustrated in FIG. 25 in that thetransmission device 101 does not have the image processing unit 113 andthat the reception device 102 has the image processing unit 141additionally.

The image processing unit 141 includes a strobe image generation unit151 and an effect processing unit 152. The image processing unit 141 issupplied with free viewpoint data from the decoding unit 132. The imageprocessing unit 141 uses the free viewpoint data from the decoding unit132 to generate as necessary a strobe image showing 3D models of asubject at a plurality of times (frames) in viewpoint images captured bythe image capturing unit 111, and performs an effect process on thestrobe image.

In other words, as with the strobe image generation unit 121 illustratedin FIG. 25, the strobe image generation unit 151 uses the free viewpointdata from the decoding unit 132 to generate a strobe image showing 3Dmodels at a plurality of times, and supplies the free viewpoint data ofthe strobe image to the effect processing unit 152.

As with the effect processing unit 122 illustrated in FIG. 25, theeffect processing unit 152 performs an effect process on the 3D modelseen in the strobe image in the free viewpoint data supplied from thestrobe image generation unit 151, and supplies the free viewpoint dataof the strobe image having undergone the effect process to the freeviewpoint image generation unit 133.

Note that the image processing unit 141 can supply the free viewpointdata from the strobe image generation unit 151 to the free viewpointimage generation unit 133 without the effect process by the effectprocessing unit 152 according to the user's operation or the like, forexample. Moreover, the image processing unit 141 can supply the freeviewpoint data from the decoding unit 132 to the free viewpoint imagegeneration unit 133 without the generation of a strobe image at thestrobe image generation unit 151 and the effect process by the effectprocessing unit 152 according to the user's operation or the like, forexample. Furthermore, the image processing unit 141 can performgeneration of a strobe image and an effect process in parallel, that is,can generate a (3D) strobe image after effect process in which an effectprocess has been performed on 3D models from free viewpoint data.

FIG. 29 is a flowchart of a first example of a process by thetransmission device 101 illustrated in FIG. 28.

In steps S151 and S152, processing similar to that in steps S111 andS112 of FIG. 26 is performed. Thereby, the free viewpoint data generatedby the free viewpoint data generation unit 112 is supplied to theencoding unit 114, and the process proceeds from step S152 to step S153.

In step S153, the encoding unit 114 sets, for example, the effect flag(FIG. 14), that is, the effect mode, the effect direction, and theeffect distance in accordance with the user's operation on thetransmission device 101 or the like, and the process proceeds to S154.

In step S154, the encoding unit 114 encodes the free viewpoint data fromthe free viewpoint data generation unit 112 to generate coded dataincluding an effect flag. Then, the encoding unit 114 supplies the codeddata to the transmission unit 115, and the process proceeds from stepS154 to step S155.

In step S155, the transmission unit 115 transmits the coded data fromthe encoding unit 114, and the process is terminated.

As described above, the transmission device 101 can generate andtransmit coded data including an effect flag set in response to theuser's operation on the transmission device 101 or the like.

FIG. 30 is a flowchart of a first example of a process by the receptiondevice 102 illustrated in FIG. 28.

in step S161, the reception unit 131 of the reception device 102receives the coded data transmitted from the transmission unit 115 ofthe transmission device 101 and supplies the coded data to the decodingunit 132, and the process proceeds to step S132.

In step S162, the decoding unit 132 decodes the coded data from thereception unit 131 and supplies free viewpoint data obtained by thedecoding and the effect flag to the image processing unit 141, and theprocess proceeds to step S163.

in step S163, the strobe image generation unit 151 of the imageprocessing unit 141 determines whether to generate a strobe image.

Here, the determination as to whether to generate a strobe image in stepS163 can be made according to, for example, the user's operation on thereception device 102.

When it is determined in step S163 that a strobe image is not to begenerated, the strobe image generation unit 151 and the effectprocessing unit 152 supply free viewpoint data to the free viewpointimage generation unit 133 without performing processing. Then, theprocess proceeds from step S163 to step S167 shipping steps S164 toS166.

Furthermore, when it is determined in step S163 that a strobe image isto be generated, the process proceeds to step S164.

In step S164, the strobe image generation unit 151 generates a strobeimage using the free viewpoint data supplied from the decoding unit 132to the image processing unit 141.

In other words, the strobe image generation unit 151 selects a pluralityof generation frames from the frames of the free viewpoint data from thedecoding unit 132. Moreover, the strobe image generation unit 151generates a strobe image showing 3D models (3D models at a plurality oftimes) of the subject shown in the plurality of generation frames.

The strobe image generation unit 151 supplies free viewpoint data of thestrobe image to the effect processing unit 152, and the process proceedsfrom step S164 to step S165.

In step S165, the effect processing unit 152 determines whether toperform an effect process on the 3D models of the strobe image of whichthe free viewpoint data is supplied from the strobe image generationunit 151. The determination as to whether to generate a strobe image instep S165 can be made according to the user's operation on the receptiondevice 102 and the like, for example.

When it is determined in step S165 that an effect process is not to beperformed, the effect processing unit 152 supplies the free viewpointdata of the strobe image from the strobe image generation unit 151 tothe free viewpoint image generation unit 133 without performing aneffect process. Then, the process proceeds from step S165 to step S167skipping step S166.

Furthermore, when it is determined in step S165 that an effect processis to be performed, the process proceeds to step S166.

In step S166, according to the effect flag supplied from the decodingunit 132 to the image processing unit 141, the effect processing unit152 performs an effect. process on the 3D models at a plurality of times(generation frames) shown in the strobe image of which the freeviewpoint data is supplied from the strobe image generation unit 151.

For example, in a case where the effect mode is set to effect mode 1,the effect direction is set to past, and the effect distance is set totime=10, the effect processing unit 152 performs an effect process totransparentize the 3D models separated by 10 seconds or more in the pastdirection from the reference 3D model in the strobe image.

Then, the effect processing u u nit 152 supplies the free viewpoint dataof the strobe image having undergone the effect process to the freeviewpoint image generation unit 133, and the process proceeds from stepS166 to step S167.

In step S167, the free viewpoint image generation unit 133 uses the freeviewpoint data from (the effect processing unit 152 of) the imageprocessing unit 141 to generate, as free viewpoint images, 2D images inwhich a three-dimensional space captured by the image capturing unit 111of the transmission device 101 is viewed from a virtual viewpoint set bythe user's operation on the reception device 102 or the like. In otherwords, for example, the free viewpoint image generation unit 133generates 2D images in which a three-dimensional space shown in a strobeimage obtained by performing an effect process on 3D models is viewedfrom a virtual viewpoint, as free viewpoint images. Then, the freeviewpoint image generation unit 133 supplies the free viewpoint image(data) to the display unit 134, and the process proceeds from step S167to step S168.

in step S168, the display unit 134 displays the free viewpoint imagesfrom the free viewpoint image generation unit 133, and the process isterminated.

By the processing described above, the reception device 102 can displaythe strobe image that shows the 3D models having undergone the effectprocess according to the effect flag set according to the user'soperation on the transmission device 101.

FIG. 31 is a flowchart of a second example of a process by thetransmission device 101 illustrated in FIG. 28.

In steps S181 and S182, processing similar to that in steps S151 andS152 of FIG. 29 is performed. Thereby, the free viewpoint data generatedby the free viewpoint data generation unit 112 is supplied to theencoding unit 114, and the process proceeds from step S182 to step S183.

In step S183, the encoding unit 114 encodes the free viewpoint data fromthe free viewpoint data generation unit 112 to generate coded data.Then, the encoding unit 114 supplies the coded data to the transmissionunit 115, and the process proceeds from step S183 to step S184.

In step S184, the transmission unit 115 transmits the coded data fromthe encoding unit 114, and the process is terminated.

As described above, in the case of FIG. 31, unlike in the case of FIG.29, the transmission device 101 generates and transmits coded data notincluding an effect flag.

FIG. 32 is a flowchart of a second example of a process by the receptiondevice 102 illustrated in FIG. 28.

In step S191, the reception unit 131 of the reception device 102receives the coded data transmitted from the transmission unit 115 ofthe transmission device 101 and supplies the coded data to the decodingunit 132, and the process proceeds to step S132.

In step S192, the decoding unit 132 decodes the coded data from thereception unit 131 and supplies free viewpoint data obtained by thedecoding to the image processing unit 141, and the process proceeds tostep S193.

In steps S193 to S195, processing similar to steps S163 to S165illustrated in FIG. 30 is performed.

Then, when it is determined in step S195 that an effect process is notto be performed, the effect processing unit 152 supplies the freeviewpoint data of the strobe image from the strobe image generation unit151 to the free viewpoint image generation unit 133 without performingan effect process. Then, the process proceeds from step S195 to stepS198 skipping steps S196 and S197.

Furthermore, when it is determined in step S195 that an effect processto be performed, the process proceeds to step S196.

In step S196, the effect processing unit 152 sets, for example, theeffect flag (FIG. 14), that is, the effect mode, the effect direction,and the effect distance in accordance with the user's operation on thereception device 102 or the like, and the process proceeds to S197.

Here, the user of the reception device 102 may feel troublesome to setindividually the effect mode, the effect direction, and the effectdistance as the effect flag. Therefore, in the effect processing unit152, there is prepared a plurality of combinations of the effect mode,the effect direction, and the effect distance as the effect flag, aseffect filters. This allows the user of the reception device 102 to, instep S196, select a desired filter among the plurality of effect filtersand set the desired filter as the effect mode, the effect direction, andthe effect distance as the effect flag.

In step S197, according to the effect flag set according to the user'soperation on the reception device 102, the effect processing unit 152performs an effect process on the 3D models at a plurality of times(generation frames) shown in the strobe image of which the freeviewpoint data is supplied from the strobe image generation unit

Then, the effect processing unit 152 supplies the free viewpoint data ofthe strobe image having undergone the effect process to the freeviewpoint image generation unit 133, and the process proceeds from stepS197 to step S198.

In steps S198 and S199, processing similar to that in steps S167 andS168 of FIG. 30 is performed and the process is terminated.

By the processing described above, the reception device 102 can displaythe strobe image that shows the 3D models having undergone the effectprocess according to the effect flag set according to the user'soperation on the reception device 102.

<Image Processing System to which the Present Technology is Applied>

FIG. 33 is a block diagram illustrating a configuration example ofanother embodiment oi an image processing system to which the presenttechnology is applied.

Note that the components corresponding to those illustrated in FIG. 1are denoted with the same reference numerals, and the descriptionthereof will be appropriately omitted below.

The image processing system illustrated in FIG. 33 includes an imagecapturing unit 11, a free viewpoint data generation unit 12, an effectprocessing unit 14, a free viewpoint image generation unit 15, and adisplay unit 16.

The image processing system illustrated in FIG. 33 is in common with theimage processing system illustrated in FIG. 1 in including an imagecapturing unit 11, a free viewpoint data generation unit 12, an effectprocessing unit 14, a free viewpoint image generation unit 15, and adisplay unit 16. However, the image processing system illustrated inFIG. 33 is different from the image processing system illustrated inFIG. 1 in that the strobe image generation unit 33 is not provided.

As described above, since the image processing system of FIG. 33 doesnot have the strobe image generation unit 33, free viewpoint data of a3D image is supplied to the effect processing unit 14 from the freeviewpoint data generation unit 12.

Then, the effect processing unit 14 performs an effect process on 3Dmodels of a subject shown in the 3D image of which the free viewpointdata is supplied from the free viewpoint data generation unit 12.

For example, when a plurality of subjects is shown in the 3D image ofwhich free viewpoint data is supplied from the free viewpoint datageneration unit 12, the effect processing unit 14 can perform an effectprocess on each of 3D models of one or more of the plurality ofsubjects.

Therefore, when the 3D image of which the free viewpoint data issupplied from the free viewpoint data generation unit 12 to the effectprocessing unit 14 is an image of contents of a soccer game, the effectprocessing unit 14 can perform an effect process on 3D models of aplurality of soccer players or a ball and one or more soccer playersnear the ball as a plurality of subjects shown in the 3D image, forexample.

As described above, the effect processing unit 14 can perform an effectprocess on one or more 3D models among 3D models of a plurality ofsubjects shown in any 3D image other than a strobe image (including all3D models of the plurality of subjects shown in the 3D image).

FIG. 34 is a flowchart of an example of a free viewpoint image displayprocess of displaying a free viewpoint image performed by the imageprocessing system illustrated in FIG. 33.

In the free viewpoint image display processing, in steps S211 and S212,the same processing as steps S11 and S12 illustrated in FIG. 2 isperformed. Accordingly, the free viewpoint data generation unit 12generates 3D models of a subject (and 3D data including a backgroundimage), for example, as free viewpoint data of the 3D image on aframe-by-frame basis. The free viewpoint data (frame by frame) of the 3Dimage is supplied from the free viewpoint data generation unit 12 to theeffect processing unit 14, and the process proceeds from step S212 tostep S213.

In step S213, the effect processing unit 14 determines whether toperform an effect process on the 3D models shown in the 3D image ofwhich the free viewpoint data is supplied from the free viewpoint datageneration unit 12. The determination as to whether to generate a strobeimage in step S217 can be made, for example, according to the user'soperation.

When it is determined in step S213 that an effect process is not to beperformed, the effect processing unit 14 supplies the free viewpointdata of the 3D image from the free viewpoint data generation unit 12 tothe free viewpoint image generation unit 15 without performing theeffect process. Then, the process proceeds from step S213 to step S215skipping step S214.

In this case, in step S215, the free viewpoint image generation unit 15generates, as free viewpoint images, 2D images in which the 3D image(free viewpoint data) from the effect processing unit 14 is viewed fromvirtual viewpoints. Then, the free viewpoint image generation unit 15supplies the free viewpoint image to the display unit 16, and theprocess proceeds from step S215 to step S216.

In step S216, the display unit 16 displays the free viewpoint imagesfrom the free viewpoint image generation unit 15. In this case, thedisplay unit 16 displays 2D images showing 3D models of the subjectviewed from virtual viewpoints (2D images in which the 3D image isviewed from virtual viewpoints).

On the other hand, when it is determined in step S213 that an effectprocess is to be performed, the process proceeds to step S214.

In step S214, the effect processing unit 14 performs an effect processon 3D models of a subject shown in the 3D image of which the freeviewpoint data is supplied from the free viewpoint data generation unit12.

Here, when one subject is shown in the 3D image, the effect processingunit 14 performs an effect process on 3D models of the one subject.Furthermore, when a plurality of subjects is shown in the 3D image, theeffect processing unit 14 can select 3D models of one or more subjectsfrom the 3D models of the plurality of subjects, and perform an effectprocess on each of the 3D models of the one or more subjects. The 3Dmodels to be subjected to the effect processing can be selected, forexample, according to the user's operation or the like.

The effect processing unit 14 supplies the 3D image showing the 3Dmodels having undergone the effect process to the free viewpoint imagegeneration unit 15, and the process proceeds from step S214 to stepS215.

In this case, in step S215, the free viewpoint image generation unit 15generates, as free viewpoint images, 2D images in which the 3D imageshowing the 3D models having undergone the effect process from theeffect processing unit 14 is viewed from virtual viewpoints. Then, thefree viewpoint image generation unit 15 supplies the free viewpointimage to the display unit 16, and the process proceeds from step S215 tostep S216.

In step S216, the display unit 16 displays the free viewpoint imagesfrom the free viewpoint image generation unit 15. In this case, thedisplay unit 16 displays 2D images (2D images having undergone theeffect process) in which an effect is applied to the 3D models of one ormore subjects viewed from the virtual viewpoints.

As described above, it is possible to provide an easily viewable imageby performing an effect process on one or more 3D models among 3D modelsof a plurality of subjects shown in a 3D image other than a strobeimage. For example, as described above, when the 3D image is an image ofcontent of a soccer game, it is possible to provide an image in which aneffect is applied to 3D models of a plurality of soccer players relatedto a goal, the ball and one or more soccer players near the ball, or theball, the soccer player making a pass, and the soccer player receivingthe pass, or the like as a plurality of subjects shown in the 3D image.

<Transmission System>

FIG. 35 is a block diagram illustrating a configuration example of atransmission system to which the image processing system illustrated inFIG. 33 is applied.

Note that the components corresponding to those illustrated in FIG. 25are denoted with the same reference numerals, and the descriptionthereof will be appropriately omitted below.

Referring to FIG. 35, the transmission system includes a transmissiondevice 101 and a reception device 102. The transmission device 101includes an image capturing unit 111, a free viewpoint data generationunit 112, an image processing unit 113, an encoding unit 114, and atransmission unit 115, and a free viewpoint image generation unit 211.The image processing unit 113 has an effect processing unit 122. Thereception device 102 includes a reception unit 131, a decoding unit 132,and a display unit 134.

Therefore, the transmission system illustrated in FIG. 35 is in commonwith the transmission system illustrated in FIG. 25, in that thetransmission system includes the transmission device 101 and thereception device 102, the transmission device 101 includes the imagecapturing unit 111, the free viewpoint data generation unit 112, theimage processing unit 113, the encoding unit 114, and the transmissionunit 115, and that the reception device 102 includes the reception unit131, the decoding unit 132, and the display unit 134.

However, the transmission system illustrated in FIG. 28 is differentfrom the transmission system illustrated in FIG. 25, in that the imageprocessing unit 113 of the transmission device 101 does not include thestrobe image generation unit 121, the transmission device 101additionally includes a free viewpoint image generation unit 211, andthe reception device 102 does not include the free viewpoint imagegeneration unit 133.

In the transmission device 101, the image capturing unit 111, the freeviewpoint data generation unit 112, the effect processing unit 122 ofthe image processing unit 113, and the free viewpoint image generationunit 211 respectively perform processes in manners similar to the imagecapturing unit 11, the free viewpoint data generation unit 12, theeffect processing unit 14, and the free viewpoint image generation unit15 illustrated in FIG. 33.

Accordingly, the free viewpoint image generation unit 211 supplies a 2Dimage in which an effect is applied to 3D models of a subject viewedfrom virtual viewpoints or a 2D image in which no effect is applied to3D models of a subject viewed from virtual viewpoints, as free viewpointimages, to the encoding unit 114.

The encoding unit 114 encodes the free viewpoint images supplied fromthe effect processing unit 122 of the image processing unit 113according to a predetermined encoding method, and supplies coded dataobtained by the encoding to the transmission unit 115. The transmissionunit 115 transmits the coded data from the encoding unit 114 by wiredcommunication or wireless communication.

In the reception device 102 (display device), the reception unit 131receives the coded data transmitted from (the transmission unit 115 of)the transmission device 101, and supplies the coded data to the decodingunit 132.

The decoding unit 132 decodes the coded data from the reception unit 131into a free viewpoint image, and supplies the free viewpoint image tothe display unit 134. The display unit 134 displays the free viewpointimage from the decoding unit 132.

The user of the reception device 102 can specify virtual viewpoints byoperating the reception device 102, for example. The reception device102 can transmit the virtual viewpoints specified by the user to thetransmission device 101. In the transmission device 101, the freeviewpoint image generation unit 211 can generate a free viewpoint imageas a 2D image viewed from the virtual viewpoint according to the virtualviewpoint from the reception device 102.

<Variations of Effect Processes>

As effect processes performed by the effect processing unit 14 (and theeffect processing units 122 and 152), various effect processes otherthan the effect processes in the effect modes 1 to 10 illustrated inFIG. 14 can be adopted.

For example, a process of changing the size of the 3D models, a processof changing the texture (texture material) of the 3D models, a processof deleting the texture of the 3D models, and the like can be adopted aseffect processes performed by the effect processing unit 14. Moreover,for example, a process of blurring the shape of 3D models (3D shapemodels), a process of changing the color of 3D models, and the like canbe adopted as effect processes performed by the effect processing unit14. For these processes for effect, as in the effect processes in effectmodes 1 to 10 illustrated in FIG. 14, the effect direction, the effectdistance, and necessary parameters can be set.

FIG. 36 is a diagram describing an effect process of changing the sizeof 3D models.

In the effect process of changing the size of 3D models, for example,the size of each of the 3D models can be reduced or enlarged withreference to a predetermined point such as the gravity center of the 3Dmodel. Here, in a case where the 3D models before size change are incontact with a plane, simply reducing or enlarging the size of the 3Dmodels with reference to the gravity centers of the 3D models mightbring the 3D models after the size change into a state of separatingfrom the plane and floating in the air or sinking in the plane.

In other words, FIG. 36 illustrates, for example, a (3D) strobe imageshowing 3D models of a soccer ball rolling on a soccer ground.

When an effect process is performed to reduce the size of the 3D modelswith reference to the gravity centers of the 3D models of the soccerball, the 3D models of the soccer ball float from the ground asillustrated in FIG. 36. As a result, the strobe image after the effectprocess becomes unnatural.

Note that, when an effect process is performed to enlarge the size of 3Dmodels with reference to the gravity centers of the 3D models of thesoccer ball, the 3D models of the soccer ball come to sink into theground. As a result, the strobe image after the effect process becomesunnatural.

Therefore, in the case of performing an effect process of changing thesize of 3D models, when the 3D models before size change are in contactwith a plane, the effect processing unit 14 can move the 3D models aftersize change such that the 3D models after size change are in contactwith the plane as illustrated in FIG. 36.

This prevents the strobe image after the effect process from becomingunnatural.

In a case where 3D models (before size change) are in contact with aplane, as described above, the 3D models of a soccer bail may be incontact with a ground, the 3D models may be in contact with a floorsurface or a table top, or the 3D models may be in contact with a wallor a ceiling.

The 3D models after size change are moved perpendicularly to the planewith which the 3D models before resizing are in contact. Furthermore, ina case where the size of the 3D models is reduced or enlarged s times,the 3D models after size change are moved such that the distance betweenthe gravity centers of the 3D models after size change and the planebecomes s times shorter or longer than the distance before size change.

<Description of a Computer to which the Present Technology is Applied>

Next, the above-described series of processes may be performed byhardware or software in a case where the series of processes isperformed by software, programs constituting the software are installedin a general-purpose computer.

FIG. 37 is a block diagram illustrating a configuration example of anembodiment of a computer in which programs for executing the series ofprocesses described above are installed.

The programs can be recorded in advance in a hard disc 905 or a ROM 903as a recording medium included in the computer.

Alternatively, the programs can be stored (recorded) in a removablerecording medium 911 driven by a drive 909. Such removable recordingmedium 911 can be provided as so-called package software. Here, examplesof the removable recording medium 911 include a flexible disc, a compactdisc read only memory (CD-ROM), a magneto optical (MO) disc, a digitalversatile disc (DVD), a magnetic disc, a semiconductor memory, and thelike.

Note that the programs may be installed in the computer from theremovable recording medium 911 as described above, or may be downloadedto the computer via a communication network or a broadcast network andinstalled in the built-in hard disc 905. In other words, for example,the programs are wirelessly transferred from a download site to thecomputer via an artificial satellite for digital satellite broadcasting,or transferred in a wired manner to the computer via a network such as alocal area network (LAN) or the Internet.

The computer contains a central processing unit (CPU) 902, and aninput/output interface 910 is connected to the CPU 902 via a bus 901.

When an instruction is input by the user operating an input unit 907 orthe like via the input/output interface 910, the CPU 902 executes aprogram stored in a read only memory (ROM) 903 accordingly.Alternatively, the CPU 902 loads a program in the hard disc 905 into arandom access memory (RAM) 904 and executes the program.

Thus, the CPU 902 performs the processing according to theabove-described flowchart or the processing performed by theconfiguration illustrated in the above-described block diagram. Then,the CPU 902 causes the processing result to be output from the outputunit 906, transmitted from toe communication unit 908, or recorded onthe hard disc 905, for example, via the input/output interface 910, asnecessary.

Note that the input unit 907 includes a keyboard, a mouse, a microphone,and the like. Furthermore, the output unit 906 includes a liquid crystaldisplay (LCD), a speaker, and the like.

Here, in the present specification, the processing performed by thecomputer according to the programs does not necessarily have to beperformed chronologically in the order described in the flowchart. Inother words, the processing performed by the computer according to theprograms includes processing executed in parallel or separately (forexample, parallel processing or processing by an object).

Furthermore, the programs may be processed by one computer (processor)or may be distributed and processed by a plurality of computers.Moreover, the programs may be transferred to a remote computer forexecution.

Moreover, in the present specification, the system means a set of aplurality of components (apparatus, modules (parts), and others), and itdoes not matter whether or not all the components are in the samehousing. Therefore, the system may be a plurality of devices housed inseparate housings and connected via a network, or may be one devicecontaining a plurality of modules in one housing.

Note that embodiments of the present technology are not limited to theforegoing embodiments but can be modified in various manners withoutdeviating from the gist of the present technology.

For example, the present technology can have a cloud computingconfiguration in which one function is shared and processedcollaboratively by a plurality of devices via a network.

Furthermore, each of the steps described in the above-describedflowchart can be executed by one device or can be shared by a pluralityof devices.

Moreover, in a case where a plurality of processes is included in onestep, the plurality of processes included in one step can be executed byone device or can be shared and executed by a plurality of devices.Furthermore, the advantageous effects described herein are mere examplesbut are not limited ones. The present technology may have any otheradvantageous effects.

Note that the present technology may be configured as described below.

<1>

An image processing device including:

an effect processing unit that performs an effect process on one or moreof a plurality of 3D models generated from a plurality of viewpointimages captured from a plurality of viewpoints; and

a generation unit that generates a 2D image in which the plurality of 3Dmodels having undergone the effect process is viewed from apredetermined viewpoint.

<2>

The image processing device according to <1>, in which the effectprocessing unit performs the effect process of transparentizing apredetermined 3D model among the plurality of 3D models.

<3>

The image processing device according to <1> or <2>, in which

the effect processing unit performs the effect process on 3D models of apredetermined subject at a plurality of times generated from theplurality of viewpoint images.

<4>

The image processing device according to any one or <1> to <3>, inwhich. the effect processing unit performs the effect process on the 3Dmodels of the predetermined subject specified by a number, distance, ortime of the 3D models with respect to one reference model among the 3Dmodels of the predetermined subject at the plurality of times.

<5>

The image processing device according to any one of <1> to <4>, in which

the effect processing unit sets the effect process to be performed onthe 3D models from a plurality of effect processes.

<6>

The image processing device according to any one of <1> to <5>, furtherincluding

a display unit that displays the 2D image.

<7>

The image processing device according to any one or <1> to <6>, in which

the 3D model includes View Independent.

<8>

The image processing device according to any one of <1> to <7>, in which

the effect processing unit performs the effect process to change a sizeof the 3D model, and

when the 3D model before the size change is in contact with a plane, theeffect processing unit moves the 3D model such that the 3D model afterthe size change is in contact with the plane.

<9>

An image processing method including:

performing an effect process on one or more of a plurality of 3D modelsgenerated from a plurality of viewpoint images captured from a pluralityof viewpoints; and

generating a 2D image in which the plurality of 3D models havingundergone the effect process is viewed from a predetermined viewpoint.

<10>

A program for causing a computer to serve as:

an effect processing unit that performs an effect process on each of aplurality of 3D models generated from a plurality of viewpoint imagescaptured from a plurality of viewpoints; and

a generation unit that generates a 2D image in which the plurality of 3Dmodels having undergone the effect process is viewed from apredetermined viewpoint.

<11>

A display device including:

a reception unit that receives a 2D image obtained by performing aneffect process on one or more of a plurality of 3D models generated froma plurality of viewpoint images captured from a plurality of viewpointsand generating the 2D image in which the plurality of 3D models havingundergone the effect process is viewed from a predetermined viewpoint;and

a display unit that displays the 2D image.

<A1>

An image processing device including:

a strobe image generation unit that generates a strobe image showing 3Dmodels of a subject at a plurality of times generated from viewpointimages of the subject from a plurality of viewpoints; and

an effect processing unit that performs an effect process on the 3Dmodels shown in the strobe image.

<A2>

The image processing device according to <A1>, in which the strobe imagegeneration unit generates the strobe image so that the 3D models on thenear side are displayed on a priority basis.

<A3>

The image processing device according to <A1> or <A2>, in which

the effect processing unit performs the effect process to transparentizethe 3D models.

<A4>

The image processing device according to any one of <A1> to <A3>, inwhich

the effect processing unit performs the effect process to cause the 3Dmodels to gradually disappear,

<A5>

The image processing device according to any one of <A1> to <A24>, inwhich

the effect processing unit performs the effect process to decrease anumber of textures of the 3D models.

<A6>

The image processing device according to any one of <A1> to <A5>, inwhich

the effect processing unit performs the effect process to erase the 3Dmodels.

<A1>

The image processing device according to any one of <A1> to <A6>, inwhich

the effect processing unit performs the effect process to decreaseluminance or saturation of the 3D models.

<A8>

The image processing device according to any one of <A1> to <A7>, inwhich

the effect processing unit performs the effect process to limit a numberof the 3D models shown in the strobe image.

<A9>

The image processing device according to any one of <A1> to <A8>, inwhich

the effect processing unit performs the effect process to decrease anumber of meshes of the 3D models.

<A10>

The image processing device according to any one of <A1> to <A9>, inwhich

the effect processing unit performs the effect process to change anexpression form of the 3D models.

<A11>

The image processing device according to any one of <A1> to <A10>, inwhich

the effect processing unit performs the effect process to change the 3Dmodels formed from polygons into mire frames.

<A12>

The image processing device according to any one of <A1> to <A11>, inwhich

the effect processing unit performs the effect process to change theexpression form of the 3D models from View Dependent to ViewIndependent.

<A13>

The image processing device according to any one of <A1> to <A12>, inwhich

the effect processing unit performs the effect process to erase the 3Dmodels with traces of the 3D models left.

<A14>

The image processing device according co any one of <A1> to <A13>, inwhich

the effect processing unit performs the effect process on the 3D modelspreceding or following a reference 3D model

at a time when the latest virtual viewpoint is set among the 3D modelsshown in the strobe image. d

<A15>

The image processing device according to any one of <A1> to <A14>, inwhich

the effect processing unit performs the effect process on the 3D modelsseparated by a predetermined number of models or more from the reference3D model at a time when the latest virtual viewpoint is set among the 3Dmodels shown in the strobe image.

<A16>

The image processing device according to any one of <A1> to <A15>, inwhich

the effect processing unit performs the effect process on the 3D modelsseparated by a predetermined distance or more from the reference 3Dmodel at a time when the latest virtual viewpoint is set among the 3Dmodels shown in the strobe image.

<A17>

The image processing device according co any one of <A1> to <A16>, inwhich

the effect processing unit performs the effect process on the 3D modelsat times separated by a predetermined time or more from the reference 3Dmodel at a time when the latest virtual viewpoint is set among the 3Dmodels shown in the strobe image.

<A18>

An image processing method including:

generating a strobe image showing 3D models of a subject at a pluralityof times generated from viewpoint images of the subject from a pluralityof viewpoints; and

performing an effect process on the 3D models shown in the strobe image.

<A19>

A program for causing a computer to serve as:

a strobe image generation unit that generates a strobe image showing 3Dmodels of a subject at a plurality of times generated from viewpointimages of the subject from a plurality of viewpoints; and

an effect processing unit that performs an effect process on the 3Dmodels shown in the strobe image.

<A20>

A transmission system including:

a transmission device that has a transmission unit that transmits aneffect flag relating to an effect process to be performed on 3D modelsin a strobe image showing the 3D models of a subject at a plurality oftimes generated from viewpoint images of the subject from a plurality ofviewpoints; and

a reception device that has

a reception unit that receives the effect flag,

a strobe image generation unit that generates the strobe image, and

an effect processing unit that performs the effect process on the 3Dmodels shown in the strobe image according to the effect flag.

<B1>

An image processing device that generates a strobe image after an effectprocess in which

the effect process is performed on 3D models shown in the strobe imageshowing the 3D models of a subject at a plurality of times generatedfrom viewpoint images of the subject from a plurality of viewpoints.

<C1>

An image processing device including:

circuitry configured to:

perform an effect process on at least one 3D model of a plurality of 3Dmodels generated from a plurality of viewpoint images captured from aplurality of viewpoints; and

generate a 2D image in which the plurality of 3D models Saving undergonethe effect process is viewed from a predetermined viewpoint.

<C2>

The image processing device according to <C1>, wherein the circuitry isfurther configured to:

perform the effect process to transparentize the at least one 3D modelof the plurality of 3D models.

<C3>

The image processing device according to <C1> or <C2>, wherein thecircuitry is further configured to:

perform the effect process on 3D models of a predetermined subject atplurality of times generated from the plurality of viewpoint images.

<C4>

The image processing device according to any one of <C1> to <C3>,wherein the circuitry is further configured to:

perform the effect process on the 3D models of the predetermined subjectspecified by a number, distance, or time of the 3D models with respectto a reference model of the plurality of 3D models of the predeterminedsubject at the plurality of time.

<C5>

The image processing device according to any one of <C1> to <C4>,wherein the circuitry is further configured to:

set the effect process to be performed on the 3D models from a pluralityof effect processes.

<C6>

The image processing device according to any one of <C1> to <C5>,

wherein the circuitry is further configured to:

initiate display of the 2D image.

<C7>

The image processing device according to any one of <C1> to <C6>,wherein

an expression form of the 3D model includes a View Independent model.

<C8>

The image processing device according to any one of <C1> to <C7>,wherein the circuitry is further configured to:

perform the effect process to change a size of the 3D model, and

move, when the 3D model prior to the size change is in contact with aplane, the 3D model such that the 3D model after the sire change is incontact with the plane.

<C9>

An image processing method including:

performing an effect process on at least one 3D model of a plurality of3D models generated from a plurality of viewpoint images captured from aplurality of viewpoints; and

generating a 2D image in which the plurality of 3D models havingundergone the effect process is viewed from a predetermined viewpoint.

<C10>

A non-transitory computer-readable medium having embodied thereon aprogram, which when executed by a computer causes the computer toexecute an image processing method, the method including:

performing an effect process on at least one 3D model of a plurality of3D models generated from a plurality of viewpoint images captured from aplurality of viewpoints; and

generating a 2D image in which the plurality of 3D models havingundergone the effect process is viewed from a predetermined viewpoint.

<C11>

A display device including:

circuitry configured to:

receive a 2D image obtained by performing an effect process on at leastone 3D model of a plurality of 3D models generated from a plurality ofviewpoint images captured from a plurality of viewpoints and generatingthe 2D image in which the plurality of 3D models having undergone theeffect process is viewed from a predetermined viewpoint; and

display the 2D image.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

REFERENCE SIGNS LIST

11 Image capturing unit

12 Free viewpoint data generation unit

13 Strobe image generation unit

14 Effect processing unit

15 Free viewpoint image generation unit

16 Display unit

101 Transmission device

102 Reception device

111 Image capturing unit

112 Free viewpoint data generation unit

113 Image processing unit

114 Encoding unit

115 Transmission device

131 Reception device

132 Decoding unit

133 Free viewpoint image generation unit

134 Display unit

141 Image processing unit

151 Strobe image generation unit

152 Effect processing unit

901 Bus

902 CPU

903 ROM

904 RAM

905 Hard disk

906 Output unit

907 Input unit

908 Communication unit

909 Drive

910 Input/output interface

911 Removable recording medium

1. An image processing device comprising: circuitry configured to:perform an effect process on at least one 3D model of a plurality of 3Dmodels generated from a plurality of viewpoint images captured from aplurality of viewpoints; generate a 2D image in which the plurality of3D models having undergone the effect process is viewed from apredetermined viewpoint.
 2. The image processing device according toclaim 1, wherein the circuitry is further configured to: perform theeffect process to transparentize the at least one 3D model of theplurality of 3D models.
 3. The image processing device according toclaim 1, wherein the circuitry is further configured to: perform theeffect process on 3D models of a predetermined subject at plurality oftimes generated from the plurality of viewpoint images.
 4. The imageprocessing device according to claim 3, wherein the circuitry is furtherconfigured to: perform the effect process on the 3D models of thepredetermined subject specified by a number, distance, or time of the 3Dmodels with respect to a reference model of the plurality of 3D modelsof the predetermined subject at the plurality of time.
 5. The imageprocessing device according to claim 1, wherein the circuitry is furtherconfigured to: set the effect process to be performed on the 3D modelsfrom a plurality of effect processes.
 6. The image processing deviceaccording to claim 1, wherein the circuitry is further configured to:initiate display of the 2D image.
 7. The image processing deviceaccording to claim 1, wherein an expression form of the 3D modelincludes a View Independent model.
 8. The image processing deviceaccording to claim 1, wherein the circuitry is further configured to:perform the effect process to change a size of the 3D model; and move,when the 3D model prior to the size change is in contact with a plane,the 3D model such that the 3D model after the size change is in contactwith the plane.
 9. An image processing method comprising: performing aneffect process on at least one 3D model of a plurality of 3D modelsgenerated from a plurality of viewpoint images captured from a pluralityof viewpoints; and generating a 2D image in which the plurality of 3Dmodels having undergone the effect process is viewed from apredetermined viewpoint.
 10. A non-transitory computer-readable mediumhaving embodied thereon a program, which when executed by a computercauses the computer to execute an image processing method, the methodcomprising: performing an effect process on at least one 3D model of aplurality of 3D models generated from a plurality of viewpoint imagescaptured from a plurality of viewpoints; and generating a 2D image inwhich the plurality of 3D models having undergone the effect process isviewed from a predetermined viewpoint.
 11. A display device comprising:circuitry configured to: receive a 2D image obtained by performing aneffect process on at least one 3D model of a plurality of 3D modelsgenerated from a plurality of viewpoint images captured from a pluralityof viewpoints and generating the 2D image in which the plurality of 3Dmodels having undergone the effect process is viewed from apredetermined viewpoint; and display the 2D image.